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The authors have endeavored to make objective determinations in selecting 
historical information that seemed most appropriate toward achieving a fair and 
equitable treatment of the subject matter presented herein. Although some data is 
in conflict with others, every effort has been made to validate content integrity by 
citing sources within footnotes and carefully inputting the referenced data, thus 
preserving original accuracy, correctness, value, sufficiency, and completeness of 
the information. While no error was knowingly permitted to remain in this 
presentation, the authors shall have no liability to any customer or other using party 
for any loss, expense, or damages (direct or indirect) including consequential, 
incidental, special, or punitive damages, lost profits or lost revenue arising out of or 
caused by any inadvertent error or omission in connection with the information 
contained herein. 

This is a work of the U.S. Government and is not subject to copyright protection 
in the United States. The published product may be reproduced and distributed in 
its entirety without further permission from GAO. However, because this work 
may contain copyrighted images or other material, permission from the copyright 
holder may be necessary if you wish to reproduce this material separately. 
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Foreword 

Cost Estimating and NASA's Changing Culture 

This paper is intended to provide a reliable methodology for those tasked with generating price tags 
on construction (CoF) and research and development (R&D) activities in the NASA performance 
world. This document consists of a collection of cost-related engineering detail and project 
fulfillment information from early agency days to the present. Accurate historical detail is the first 
place to start when determining improved methodologies for future cost and schedule estimating. 
This paper contains a beneficial proposed cost estimating method for arriving at more reliable 
numbers for future submits. 

When comparing current cost and schedule methods with earlier cost and schedule approaches, it 
became apparent that NASA's organizational performance paradigm has morphed. Mission 
fulfillment speed has slowed and cost calculating factors have increased in 21st Century space 
exploration. Two reasons stand out: 

1) The rapid development mindset in the "space race" territory of yesteryear was 
supported by "frontier risk" thinking. Risk aversion at all costs was not yet a NASA 
moniker. Back then, an "acceptable risk level" was supported as a defensible reality. 

This "allowable hazard" paradigm is not accepted as a mission philosophy today. 

Slower development schedules have replaced earlier more aggressive performance 
timelines. Expanded regulatory constraints in the area of safety and mission 
assurance have increased engineering redundancy processes which, in turn, push 
fulfillment schedules to the right. The previous "can do" bravado has been 
supplanted by a "can do with caveats" culture. The "zero failure" standard has 
resulted in a far more complex mission readiness protocol when standing up a new 
space program. 

2) NASA of today has multiple contracting tentacles under its current business 
umbrella. Contracting requirements in today's CoF and R&D worlds are more 
complex which inevitably slows procurement and sub-development efforts. 

Regulatory constraints further complicate the systems engineering world. 

Streamlined development in former times meant beating the calendar. Streamlined 
development today usually means complying with a myriad of checklists and trying to 
tie down highly volatile baselines while still demonstrating visible progress toward 
project end-goals. Concurrent development of back-up scenarios that can, if 
necessary, be brought on line with minimal cost and schedule impact inevitably 
slows the overall timeline. Calendar targets (though still acknowledged as important 
in the NASA planning world) have become subordinate to an expanded landscape of 
engineering, organizational, and regulatory constraints, protocols and procedures. 

We leave the efficacy of the "old" and "new" development paradigms within NASA to the 
judgment of the reader. 
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NASA Credibility Hangs in the Balance 


i 

M. GAO 

^AtcounUMIIty » Intoytt, ■ 

United States General Accounting Office 
Washington, D.C. 20548 


"NASA has yet to 
implement a well-defined 
process for estimating the 
cost of its programs — a 
weakness we and NASA’s 
Inspector General have 
repeatedly reported.” 

See, for example, GAO-04-1 18; GAO-04- 
255; GAO-03 114; U.S. General Accounting 
Office, Space Station: Actions Under Way to 
Manage Cost, but Significant Challenges 
Remain, 

GAO-02-735 (Washington, D.C.: July 17, 
2002); NASA Program Costs: Space 
Missions Require Substantially More 
Funding Than Initially Estimated, 

GAO/N SI AD-93-9 7 (Washington, D.C.: Dec. 
31, 1992); and NASA Office of Inspector 
General, Final Management Letter on 
Failures in Cost Estimating and Risk 
Management Weaknesses in Prior Space 
Launch Initiative Assignment Numbers A-01- 
049-01 and A-01 -049-02, IG-03-023 
(Washington, D.C.: Sept. 29, 2003). 


May 28, 2004 

The Honorable Sherwood L Boehlert 
Chairman 

The Honorable liar! Gordon 
Ranking Minority Member 
Committee on Science 
House of Representatives 

The lack of reliable financial and |>erfonnanee Information has posed 
significant challenges to the National Aeronautics and Space 
Administrations (NASA) ability to manage its largest and most costly 
programs effectively. For nearly 15 years, NASA contract management has 
been on GA( Ys high-risk list — due in part to NASA’s inability to collect, 
maintain, and ro|M>rt the full cost of its programs and projects. 1 Without 
such information. NASA has consistently developed unrealistic cost and 
schedule estimates, whic h, at least in part, are reflected in the cost growth 
ami schedule increases in many of its programs. 
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NASA's Joint Confidence Level Paradox: 

A History of Denial 

Glenn Butts, CCC, MRICS, CGC, CMC, CFC 
Kent Linton, Senior EVMS Manager - Cost Estimating - SAIC/Craig Technologies 

Abstract: 

NASA purports to seek joint probable cost and schedule reality for our programs - yet we 
intentionally omit discemable risks from our comprehensive analyses studies. 
Underestimating risk vicissitudes (and consequently under-shooting cost and schedule 
projections) is a well verified NASA phenomenon. However, NASA rarely acknowledges 
publicly this pattern of aiming high and shooting low. Cost and schedule estimating 
candidness has proven difficult to come by. NASA should actively pursue a course that will 
change our prevailing reluctance to "tell it like it is in the cost and schedule world." The goal 
of this paper is to be a catalyst in facilitating that change. 

In this publication, we will examine historical evidence relative to underestimating both costs 
and schedules for long-term NASA programs and other large scale projects. We will also 
look at how skewed confidence levels resulting from ultra-optimistic estimates can jeopardize 
program continuance. Furthermore, this study will examine detail accuracies and discuss 
methods of risk quantification. If the presentation seems to become too hard hitting at times, 
it is because we have followed the advice of Winston Churchill when he counseled: "If you 
have an important point to make, don't try to be subtle or clever. Use a pile driver. Hit the 
point once. Then come back and hit it again. Then hit it a third time - a tremendous whack. " 

The presentation approach we have chosen for this paper was calculated to win support for 
the recommendations presented herein and is ultimately aimed at achieving endorsement 
from NASA's top decision makers in an effort to reverse the practice of submitting historically 
low cost projections and overly optimistic schedule milestones. Every effort has been made 
to reflect NASA's actual business reality - i.e. the way NASA builds programs, fulfills its 
objectives, and achieves its goals. Relevant case studies and examples will be provided 
going back in history even before NASA's organizational genesis. These examples reveal 
that the cost-schedule paradox is a morass that has chronically plagued the financial 
estimating community and construction schedulers throughout a variety of work venues for 
the past 200 years. 

The primary objective of this presentation is to introduce to the NASA business community 
what the author's feel is the optimum hybrid model for accurately estimating cost and 
schedule reality in complex science and engineering environments where maturing 
technologies are present. We refer to this heuristic model as the Joint Confidence Level - 
Probabilistic Calculator (JCL-PC). The JCL-PC approach merges years of professional 
analysis into a tool which compensates mathematically for what human monitoring cannot 
determine, foreshadowing the need for illusive fiscal requirements and schedule realities so 
essential to reliable cost estimating output. 

Key Words: Joint Confidence Level (JCL), Cost Overrun, Cost Growth, Schedule Slip, 
Estimate Confidence Level, JCL-PC, Boondoggle, Monte Carlo, Probabilistic Estimates, Risk 
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A Paradox at NASA 

A paradox is a statement that contradicts itself. We at NASA purport to seek the joint probable cost 
and schedule for our programs yet we exclude many credible risks from our analyses. To paraphrase 
Occam's razor: "The simplest explanation is often the correct one" leads us to two possible conclusions: 

• Conclusion 1: We really are not seeking probable cost and schedule accuracy, but are primarily 
concerned with keeping programs viable and fundable as long as possible. 

• Conclusion 2: The variables are too complex, unquantifiable, and incalculable. 

With respect to conclusion #1, an unidentified European civil servant has observed: 


"You will (as a planner) know the real costs. You will realize that the budget is too low 
but it is difficult to pass such a message to the managers and the private actors. They 
know that high costs reduce the chances of funding." 

Author Unknown 


If this form of malfeasance is real - though never spoken - then we must all sadly confess that ethics 
training has failed us at the very foundation of our professional identity. We would hope that 
expediency would never cloak our responsibility to tell it like it is. If, however, staying afloat in this 
"half truth" way has been done in the past, we can certainly resolve that it will NOT be resorted to in the 
future. We owe this to ourselves and to the worthy reputation that NASA has won during its matchless 
career as an agency. 

With respect to conclusion #2, we submit that although the science/art of cost and schedule estimating is 
multi-layered and multi-dimensional, outcomes ARE more discemable than current cost and schedule 
methodologies will allow. Joint Confidence Level (JCL) analysis is being sold as a panacea for solving 
the ills that have dogged the cost estimating community. We hold that JCL, as it is currently being 
implemented, will not deliver reliable results currently being advertised. Something more is needed! 

This paper presents a case for what that additional "something more" is. We acknowledge that 
probability analysis IS indeed complex, but it does have bounds and dimensions that are quantifiable and 
which will give up steering secrets when probed with the right tools. Our approach engages additional 
types of assessment techniques. Starting with the best estimates that our subject experts can provide, we 
1) perform distributive curve statistical analysis, together with 2) project phase evaluation in such a way 
as to predict with substantially greater precision the final cost and schedule outcomes of new 
engineering efforts. We refer to our tool as the Joint Confidence Level - Probabilistic Calculator or 
JCL-PC . We are confident that implementing the JCL-PC will assist NASA in achieving the long 
sought for accuracy in its budget creation process and funding profile submissions. The need for the 
JCL-PC will be defended in the following pages of this paper which assembles evidence and statistics 
compiled from many decades of historical recordkeeping. 

What is a JCL? 

Simply stated. Joint Confidence Level (JCL) is the output data provided by a probabilistic Monte Carlo 
calculation, expressed as a percentage. Example, we are 50% confident that we can achieve our stated 
goal(s) by our stated completion date, given our current estimated funding level. This is a confidence 
level declaration that only looked at cost probabilities. Much has been written and is readily available 
on the subject of estimate confidence levels for those desiring to enhance their understanding. 
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Biases Defined 

"Bias" refers to any consistent tendency based on human perspective for estimates to be lower or higher 
than actual cost outcomes. "Dispersion" relates to the probability (likelihood) that an individual estimate 
may differ from its actual cost, even after the bias is taken into account. 



Business Case Contract Award Works Completion 


Figure 1: Optimism Bias Chart 

Optimism bias is defined as a measure of the extent to which actual project costs (both capital and 
operating) and actual project duration, or the time it takes from contract award to benefit delivery, 
exceed the estimated cost and time data. Optimism bias is a measure of the degree by which the actual 
benefits delivered by a project fall short of the benefits that were estimated. Optimism bias means 
project achievements fall short of expectations. If a cost estimator has an optimism bias, they will 
underestimate project costs and completion schedules. Possessing an anti -optimism bias means having 
the ability to determine (in advance) with a meaningful measure of accuracy how events and costs will 
actually come to pass. Through the use of various tools (e.g. deductive reasoning, mathematical analysis 
and other outcome adjustment techniques) estimates can be made to turn out close to actuals when all is 
said and done. To a cost estimator or budget analyst this is nirvana. 

Early estimates of important parameters are generally inaccurate in two respects. Such estimates are 
almost always biased toward optimistic outcomes. Second, in addition to this natural positive bias, 
subjective errors in estimating cost calculations reveal large variations or dispersions. 1 

This paper will examine historical program cost and schedule accuracies (or lack thereof) and discuss 
several methods of risk quantification (noting limitations of each) and then introduce a methodology that 
will place the estimator on much more credible ground in providing trustworthy estimates. It must be 
stated that historical perspective is essential to understanding the nature of our challenge. NASA's way 
of doing its cost and schedule analysis in the past must give way to a new frontier. That frontier includes 
a hybrid model for NASA to use in calculating with greater precision its joint cost and schedule 
confidence levels and estimates. 

"Scientists tend to downplay costs early to convince NASA that their project is cheaper than 
someone else's. Later, once NASA commits and the money is being spent, more bucks are 
needed • so NASA spends more instead of canceling the project." 

Mike Griffin - NASA Administrator 


1 RAND Predictability of the Costs, Time, and Success of Development, 1959 
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The Importance of Determining "Project Phase" 

In addition to resolving the problematic estimating factor of optimism bias, there is one more variable 
that must be dealt with in order to achieve reliable estimates. Determining where the "program/project" 
sits on the completion timeline is tantamount. Put another way; accurately identifying what stage the 
project is in is essential to providing reliable estimates. When estimates are created early in the process, 
the estimator can only be guided by existing design plans and schedules. As development proceeds, 
initial designs and plans are often changed. This happens because of unforeseen technical difficulties 
that will prevent meeting performance requirements or to opt for tradeoffs where the net effect is to 
increase end-product performance. 

For example, on the Delta V program, the operational facilities were being constructed concurrently 
with the rocket. When the new RS68 engine was finally tested, it did not meet the expected 
performance criteria, possibly due to optimism bias. The solution meant obtaining additional fuel 
capacity which required an increase in the diameter of the propellant tanks. This single change 
reverberated down through the program and drove "cause and effect" impacts on the rocket, facilities, 
and ground support equipment. Taken together, there was serious program cost and schedule impact. 


This type of error is not the exception - but the rule. The paradox that remains to be solved can 
be framed into the following question: How do we estimate something we don 7 yet know we 
need? A variation of this question is: How do we estimate requirements which management is 
not yet prepared to acknowledge? When "deferred or unrecognized requirements" are finally 
clear enough to deal with, the Change Requests (CR's) initiated inevitably affect cost and 
schedule in a problematic way. Too often (by the end of the project) final costs incurred were 
not based on the initial BOE, thus making it impossible to reconcile final costs with initial 
estimates. When the real costs eventually come home to roost and the full extent of project 
growth can be clearly seen, the estimator rightfully states that what was actually built - was 
NOT what was originally estimated. 


The phenomenon of "underestimating" risk and consequently cost and schedule is not new or unique to 
NASA. This cost-schedule morass has existed for at least 200 years of recorded history. In 1 869, the 
Suez Canal’s cost overrun was 1,900%. Claire Brown 2 observed that prior to World War II, "virtually 
no government agency had to account for its costs or budgets." A plethora of studies have sought the 
reasons for the occurrence of such sizeable fiscal cost differentials. Understanding cost estimating 
infidelity has been an illusive pursuit due to the challenge of obtaining valid information. Nevertheless, 
it is meaningful that many experts still find agreement on the causes. To date, despite numerous 
attempts to eliminate cost and schedule over-runs, the trend of cost/schedule unreliability continues. 

"The further backward you look, the further forward you can see." 

Winston Churchill 


"Enough material has been written on the subject of cost growth during the last ten years to 
fill a minuteman silo. Unfortunately, cost growth is STILL with us." 

Witness to House Armed Services Committee (HASC) 1982 


2 An Overview of the Cost Accounting Practices Used for U.S. Defense Weapon Systems 1982 
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In many cases, project leaders or senior decision makers do not want a rigorous objective cost risk 
analysis because of the possibility that candid findings could cause a favored project to be canceled. 3 
The thinking is that if true costs are known beforehand, many projects would not be undertaken at all 
since they will not clear cost benefit analysis hurdles. The "play-it-safe-notion" often prevails - meaning 
that even if high overruns occur down the road, many project/program owners will be trapped into 
"program continuance" by the sunk cost principle. That is to say that during project execution, when 
cost overrun becomes visible, termination becomes unacceptable - and even though continuation will 
not prove to be economical, the logical decision will still be to proceed forward to completion. 

Large Program History 

Although there are volumes of information on the cost and schedule of various NASA projects, we will 
cite a few of the credible studies to date and briefly examine the best available data in an effort to ground 
our discussion and place any extrapolated conclusions on solid footing. NASA and DoD projects 
explore achievements at or near the margin of technical feasibility and often employ cost plus contracts 
which carry very little penalty for under-stating costs. Hence, contractor firms have a decided incentive 
to choose the way of under-estimation in order to get the contracts. 

We can predict with high likelihood that program costs will increase and schedules will slip as projects 
go forward. The performance equation reveals that both the cost of completion and the time estimated 
for completion will increase. Cost escalations of 100% and more are not uncommon - even increases of 
up to 200% occur more commonly than realized. Schedule growth is also common, the amount seems 
closely correlated to project type. 


"Those who do not learn from the past are condemned to repeat it." 

George Santayana 


"There's an unwillingness to recognize that there has to be a real contingency, a large 
contingency, built into any experimental program." 

W. M Allen - President Boeing - 1964 

Assembling reliable historical data containing adequate detail has been more difficult than might be 
expected. Apparently, the majority of the researchers on the cost-schedule landscape have faced the 
barrier of insufficient apples to apples detail. Some projects show contingencies - some don’t. Still 
other projects hide contingency for fear of being removed by the budget axe. Also, many projects bury 
their initial estimates and publish revisionist history as though it were pristine original data so they don’t 
look so bad in retrospect. 

Another key area of concern is the moving target of yearly inflation and cost escalation. Do the 
historical budget numbers show base year prices or were the out-years adjusted by some inflation rate 
tied to then current year dollars? Are the available numbers static with base year rates projected out over 
the life of the program? These are often unknown - making precision comparisons difficult. Perhaps it 
was the NASA environment that gave rise to the tongue-in-cheek humor that spawned the following 
quote: 


3 RAND Impossible certainty : cost risk analysis for Air Force systems / Mark V. Arena 2006 


Page 12 


4/17/2009 


NASA 's Joint Confidence Level Paradox - A History of Denial 


2009 Cost Estimating Symposium 


"It's tough to make predictions, especially about the future." 

Niels Bohr & later Yogi Berra 


Anticipating the Improbable 

Las Vegas casinos spend hundreds of millions of dollars on risk management, including high tech 
surveillance systems, and Monte Carlo analysis. Yet the four largest losses casinos have incurred or 
narrowly avoided fell completely outside their sophisticated models . 

• They lost $100 million when Roy Horn was mauled by a tiger at MGM Mirage. The tiger was 
raised by the performer and even slept in his bedroom. The casino had considered that the 
tiger may jump into the crowd, but they never thought to insure against what happened. 

• A construction worker was injured during the construction of a hotel annex. He was so 
offended at the settlement offer that he attempted to dynamite the casino. 

• Casinos are required to file a special form with the IRS documenting a gamblers profit if it 
exceeds a certain amount. The employee who was supposed to mail the forms hid them (for 
unexplainable reasons) in boxes under his desk. This went unnoticed for years before being 
discovered. The casino nearly lost their gambling license, and had to pay a huge fine. 

• In 1993, Steve Wynn’s daughter was kidnapped. He raided the casino coffers to obtain the 
ransom, at the risk of losing their gambling license. 

The dollar value of these unknown improbable events exceeds the value of the known & modeled risks 
by 1,000 to 1, yet there was no allowance for any kind of unexpected impact in analysis. 

America's First Aircraft Program 

In 1 898, in spite of the “conventional wisdom” of the day that heavier-than-air flying machines were 
impossible to construct. Dr. Samuel P. Langley was actively engaged in developing the first human 
airplane. 

By all reasonable criteria. Dr. Langley was the best and brightest program manager we had. In public 
stature and prestige, Langley was the most prominent scientist in the United States. Dr. Langley was a 
brilliant visionary who dedicated the later years of his life to the pursuit of inventing the airplane. After 
becoming Secretary of the Smithsonian in 1 886, he immediately instituted a flight research program 
using Smithsonian’s resources. In 1896, following ten years of sustained experimentation, Langley 
finally proved that human flight was achievable when his 16-foot unmanned flying machine (powered 
by a steam engine) flew nearly a mile. 4 5 

Then in 1 898, Langley succeeded in convincing the Department of War to fund a $50,000 program to 
take his experimental research to the next level. He hired a full-time staff of 10. Langley told the War 
Department 6 with “confidence” that “the machine will be completely built and ready for trial within a 
year” - meaning by 1899. As it played out, Langley would spend the entire $50,000 provided by the 
War Department, run out of money, and then raise another estimated $20,000 from other sources and 
work for five more years before attempting his first test flight. 


4 The Black Swan: The Impact of the Highly Improbable Nassim Nicholas Taleb 2007 

5 To Conquer the Air, James Tobin 

6 Langley Scrapbook, 1897-98, RU 7003, Smithsonian Institution Archives, “Extract from Proceeding of Board of Ordnance and Fortification” 
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Figure 2: - Langley's airplane - 1903 


Langley made two attempts. The first 
attempt failed on October 7, 1903 but 
the plane incurred only minor damage. 
The second attempt failed on December 
8, 1903 and altogether destroyed the 
plane. After the second failure, the 
project was lampooned. Both the media 
and Congress harshly attacked 
“Langley’s Folly.” It was evident the 
masses still believed practical human 
flight was impossible and was a waste 
of public funds. 


For over 17 years Dr. Samuel Langley had spent more than $70,000 trying to invent a powered airplane 
but had subsequently failed. The Department of War canceled the program. The Smithsonian’s regents 
forbid Langley from continuing his research. A statement from the Department of War officially 
concluded: “We are still far from the ultimate goal of human flight.” 


Yet ironically, nine days later on December 17, 1903, a brother team from Ohio, Orville & Wilbur 
Wright, (neither of whom had graduated from high school) demonstrated what most thought was 
impossible - a manned flight on a flying machine. They had invented the first airplane. What made this 
more remarkable was the fact that they had accomplished this in their spare time - as a hobby. Though it 
could rightly be considered an "obsession" with them, by working part-time over four years, the Wright 
Brothers had succeeded, spending an estimated $1,000. 7 Some lessons learned from this early example 
include the glaringly obvious point that largess ($70,000 and a staff of helpers) was not able to 
accomplish what streamlined private initiative was able to do at 1/70* the cost. 


The Famous Manhattan Project 



The original estimate for the Manhattan 
Project was $148 Million. The program 
started in 1942, and was expected to 
conclude in 1944. The final cost was $2.2 
Billion (15 times the original projected 
cost). It did not finish up until 1946 x - a 
200% schedule overrun. The geo-political 
need drove the pursuit almost totally back 
then and budget accuracy did not have its 
feet held to the fire as it does today. 
Nevertheless, the specter of cost and 
schedule was proving to be a formidable 
foe even then. 

Figure 3: -FAT MAN - The Manhattan Project's 
first atomic device cost nearly 15 times more than 
the original estimate. 


7 The Vision for Space Exploration and the Retirement of the Baby Boomers, The Space Review 

8 Richard G. Hewlett and Oscar E. Anderson, Jr., A History of the United States Atomic Energy Commission: The New World, 1939/1946, Volume I, 
(University Park, PA: The Pennsylvania State University Press, 1962 
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Eurofighter 

The Eurofighter started in the early 1980s, funded by United Kingdom, Germany, Italy, and Spain. The 
total program costs were projected to be $20 billion and the date of first service was to occur in 1997. 
Today, after two decades of technical problems and unforeseen expenses, the Eurofighter is yet to be 
deployed, and the total costs are in the $45 billion range 9 . 


RAND Corporation Studies and the DoD 

The following excerpts from an Air Force study done 50 years ago titled: "Predictability of the Costs, 
Time, and Success of Development" axe revealing: 

"To a large extent, the differences which arise do so over the question of the "extent" of the 
uncertainty in development - over questions like, "Are estimates of cost of production likely 
to be off by 25 percent or 300 percent?" ... In this paper, we present the results of some 
recent research into the extent and nature of the uncertainty in new developments. 

1) "Early estimates of important parameters are usually quite inaccurate . . . such 
estimates are strongly "biased" toward over-optimism . . . 

2) "The accuracy of estimates is a function of the stage of development, i.e. estimates 
improve as development of the item progresses . . . 

The vehicle types included in the study referred to above were fighters, bombers, tankers, and missiles 
of each of the following types: air-to-air missiles, surface-to-air missiles, air-to surface missiles, and 
surface-to-surface missiles. 
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Figure 4: - Total unadjusted 1959 cost factor increase in average cumulative cost of production 10 - Factors are ratio of latest 
available estimate, or actual costs where available, compared to earliest available estimates. In some cases the earliest 
estimate was not really an "early" estimate but was made later in the development process. 


9 Harvard Business Review Delusions of Success: How Optimism Undermines Executives’ Decisions Dan Lovallo, Daniel Kahneman 

10 RAND Predictability of Costs, Time and Success of Development 1959 
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It will be seen from the above chart, that the mean error factor for missiles is far greater (17.1) than for 
any of the different models of aircraft. Missiles fall within that category mentioned above as requiring 
more "technological advance" and hence are subject to a more aggressive estimating error. It's logical 
therefore; that NASA's space exploration mission pushes the highest limits of cost/schedule uncertainty. 


Fighters Factors 
A B 

1 3.9 4 

2 2.6 2.5 

3 2.0 2 

4 1.5 1.5 

5 1.7 2.1 

6 1.2 1.2 

7 1.0 0.8 

8 1.0 1 
9 1.1 0.6 


Bombers Factors 
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Cargos 

and 

Tankers Factors 
A B 

1 1.4 1.6 

2 1.5 1.5 

3 1.0 0.9 

4 1.0 0.8 


Missies Factors 
A B 

1 14.7 6.4 

2 9.4 6.0 

3 4.4 2.7 

4 7.2 7.1 

5 1.5 1.3 
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Figure 5: - Total adjusted 1959 Factor increase in average cumulative cost of production 11 -Adjustments were made for 
escalation, disparities between actual quantities, and planned quantities. Two sets of adjustments, by two estimators are 
included, since discretion and judgment is required for adjustments. 

Even with an "adjusted" look at the numbers, the difference in the average error factor value among the 
different classes of equipment is revealing. The smallest cost increases occurred in the Cargo and 
Tanker Class 1.2 times original estimate, with the largest increases still showing up in the Missile Class 
with the final costs 6.4 times the initial estimate. 


The explanation is that the size of the error in estimate is 1) a function of the stage of development of the 
subject article and/or 2) the magnitude of the advance being sought in that subject area. Performance 
gains for new cargo and tanker aircraft (such as range, speed, and even payload) are usually less 
dramatic when compared to what has already been achieved in other air-craft, particularly bombers. 
During the period covered by most of these estimates, missile development encompassed what was in 
many respects a new and radically different technology. Outcome was expected to show an order of 
magnitude in improvement over anything previously achieved. The large optimism bias and the 
resulting variance mirror the study's conclusions (i.e. large technological jumps vs smaller evolutionary 
efforts translate into greater cost.) 
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Figure 6: - Schedule Slip Factor accordingly ranked by the gauge of technological advance from previous generation — 
Technological advance determination was subjectively categorized by technical experts. 


11 RAND Predictability of Costs, Time and Success of Development 1959 
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Among the systems that were pioneering large technological advances, the first estimates became 
available relatively late in the program. These later estimates contain better information than was 
generally the case with "early" estimates based only on incremental evolution production-type 
projects. 
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Figure 7: - Comparison of schedule slip factor vs. project phase status for 22 aircraft and missile projects. 12 

This data also correlates with a 1962 Harvard Weapons Acquisition Research Project by Dr. Frederic 
M. Scherer that examined 12 major defense projects with a strong emphasis on performance at the 
expense of cost and schedule. Real final costs averaged seven times the initial estimate and schedules 
took 36% longer than originally planned. The last 10 percent of performance generated one-third of 
the total cost and two-thirds of the documented problems. 
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Figure 8: - The cost growth factor vs technological development from a prior system . 13 Arrows indicate direction of changes 
observed in average errors and standard deviations, in looking across the rows and down the columns, is consistent or at 
variance with the double hypothesis. Solid arrows run in direction predicted by hypothesis; dashed arrows do not. It can be 
seen that 9 out of 12 arrows point in "correct" direction and 3 do not. However, only the encircled arrows indicate 
statistically significant changes - and each of these 5 statistically significant arrows consistently point in the right direction. 
Time represents project phase. 
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12 RAND The Decision Making Problem in Development 
,} RAND The Decision Making Problem in Development 
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The "Predictability of the Costs, Time, and Success of Development" Study concludes with the 
following: 

"The data presented above is neither as comprehensive nor as unambiguous as one would 
like . . . Cost increases on the order of 200 to 300 per cent and extensions of development 
time by 1/3 to 1/2 are not the exception - but the rule . . . The optimistic bias is not hard to 
understand. Contractors are anxious to have their proposals accepted by the military, and 
the military itself is anxious to have proposals supported by the Department of Defense and 
Congress. The incentive to make optimistic estimates is thus very strong. On the other 
hand, the contractual penalties for having been overoptimistic are generally small. 

Therefore there have been great uncertainties involved in all of their planning and perhaps 
especially in the R&D area. Errors were bound to be large on the average. The real 
problem is to explain why so large a portion of the error shows up statistically as bias rather 
than variance. " 

"The variability in size of the errors observed in individual cases stems from two sources. 

One is just the basic uncertainty that characterizes ajj development work. The other is the 
difference in technological advance sought in different systems .. .the business of making 
decisions is one that calls for shrewd judgments, a large measure of skepticism, and a real 
appreciation (for) the nature of the problem. Steps can and should be taken to improve the 
estimates, especially to remove the bias. But even if this is done, significant uncertainties 
remain to harass the analyst and the decision maker"" 14 

In another RAND study, cost trends for 285 different models of aircraft were examined. For every type 
of aircraft (patrol, cargo, trainer, bomber, attack, fighter, and electronic warfare) annual unit cost 
escalation rates in the past quarter century have exceeded common inflation indices. These include the 
Consumer Price Index, the Department of Defense procurement deflator, and the Gross Domestic 
Product deflator. This trend is true whether cost escalation is measured using either procurement cost or 
flyaway cost. Patterns of cost escalation differed by aircraft - some showed cost improvement over 
time, while others steadily increased. It is important to note that RAND recognizes a 2.2% annual 
productivity increase, but has found other factors that far exceed this 15 
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Figure 9: - RAND Derived Contributors to Price Escalation Jrom the F-15A 1975 - to the F22A (2005) 



14 RAND Predictability of Costs, Time and Success of Development 1959 

15 RAND Why has the cost of fixed-wing aircraft risen? : a macroscopic examination of the trends in U.S. military aircraft costs over the past several decades / 
Mark V. Arena .2008 
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In the year 2054, the entire defense budget will purchase just one aircraft. The aircraft will 
have to be shared by the Air Force and Navy 3 'days per week except for leap year, when 
it will be made available to the Marines for the extra day." 

Norman Augustine - Aircraft Industry Executive 


Norman Augustine made this facetious prediction based on costs for individual aircraft which have 
grown by a factor of four every decade, with increases more closely related to time than capabilities. 
Trends persist across time and weapon systems. Aircraft unit costs have typically increased from 7 to 12 
percent annually. The unit costs for the F- 15 increased from $1 1.9 million in 1974 (as measured in then 
year dollars) to $54.0 million in 2000. 

DOD Programs Cost Growth Analysis 

A RAND review of Department of Defense (DoD) programs examines their history of cost growth 16 . 
Analysis of the data contained in Selected Acquisition Reports (SARs) from the late 1960s to 2004 
shows the average total cost growth factor for Major Defense Acquisition Programs (MDAPs) 
completed was 46% for all program types. This percentage was calculated by comparing program final 
costs to estimates published at Milestone phase B (MS-B) when the program was approved. It is 
important to note that most other historical cost growth comparisons listed in this section were at MS-A 
( initial program estimates). The same comparison at MS-C (when the program was approved for 
production) reveals that cost growth even in later stages had not been eliminated. In fact, it averaged 
about 16% from the MS-C decision point when the final estimate was compared to respective estimates 
at each milestone. This study revealed a systematic bias toward underestimating space systems cost , 
which was higher than cost growth for other weapons system types included in the full analysis. 
Further, the cost growth bias doesn’t disappear until % of the way between MS B and the end of 
production. The highest cost growth always occurred in the development phase, with average increases 
approaching 60% across all project types. 

The DoD study used two primary cost growth categories: 

• ERRORS - defined as inaccurate initial estimates of overall cost and schedule reflecting 
technology development to accomplish the original work scope and to meet original capabilities 
as defined at MS-B. 

• DECISIONS - defined as program changes within the control of an entity of authority such as 
the program office, SMC, Air Force, OSD, Congress, or the President. 

A companion study on the trend of cost growth during the past three decades also showed that 
development cost growth for just the past 30 year period had not improved. In fact, variability had 
actually increased, despite many attempts to reform the acquisition process 17 . The RAND study also 
showed no correlation with total program cost and cost growth. But it did reveal a substantial 
correlation to program duration - with twenty year programs experiencing almost twice the cost growth 
of ten year programs. 


16 RAND Improving the cost estimation of space systems : past lessons and future recommendations / Obaid Younossi . 2008 

17 RAND Is weapon system cost growth increasing? : a quantitative assessment of completed and ongoing programs / Obaid Younossi .2007 
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Review Status 

N 

Mean 

(Standard Deviation) 

Median 

All programs 

76 

1.45 (0.80) 

1.22 

Aircraft 

15 

1.16 (0.16) 

1.13 

Cruise missiles 

5 

1.75 (0.95) 

1.43 

Electronic aircraft 

5 

1.59 (0.31) 

1.65 

Electronics 

19 

1.20 (0.22) 

1.22 

Helicopters 

8 

1.92 (1.48) 

1.58 

Launch vehicles 

3 

1.91 (1.53) 

1.15 

Missiles 

14 

1.50 (1.04) 

1.30 

Other 

1 

1.25 

1.25 

Satellite 

3 

1.64 (0.50) 

1.88 

Vehicles 

3 

1.81 (1.06) 

1.21 


Figure 10 - RAND Average Development Cost Growth Factor Five Years After Milestone B, by Program Type. 

The values of 1.64 and 1.91 correspond to satellites and launch vehicles respectively. Similarly, median 
values are between 1.13 and 1.88, and the highest value of 1.88 corresponds to the DCGF of satellites. 
A 2005 RAND study 18 on the same subject indicates the best fit for DoD historical growth is a 
lognormal distribution - which means there is rarely if ever an occurrence of cost under-run and almost 
always an occurrence of cost overrun with the potential of having some very high cost overruns. 


Recent Trends Affecting Cost Escalation in DoD 


The real cost escalation, beyond inflation and performance growth , for military aircraft is about 3 
percent per year. Although design improvements may explain some increases, it is remarkable that other 
advances like 1) more efficient production methods, including computer aided design and 
manufacturing, 2) microminiaturization of components, and 3) the employment of greater computing 
power, have not reduced costs nor even held things level. 
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Figure 11: - RAND Average annual cost escalation for aircraft and inflation indices 1974-2005 -Excludes 2005-2008 
Commodity Price Spikes. 19 


18 RAND Toward a Cost Rick Estimating Policy 3-16-05 

19 RAND Why has the cost of fixed-wing aircraft risen? : a macroscopic examination of the trends in U.S. military aircraft costs over the past several decades / 
Mark V. Arena .2008 


Page 20 


4/17/2009 


NASA 's Joint Confidence Level Paradox - A History of Denial 


2009 Cost Estimating Symposium 


There appears to be a number of reasons for this including 1) steadily increasing requirements, 2) using 
escalation indexes that do not reflect reality, 3) consolidation in the aircraft industry which reduced 
competition and increased profits & G&A percentages, 4) increased material and parts cost, 5) the 
utilization of more expensive equipment for manufacturing, and 6) more composite materials being used 
in the manufacturing process. In the 1940’s there were 16 prime military contractors. Today there are 
two to five depending on the specific technological area in question. As a result of the consolidation in 
the defense and aerospace industries, debt to equity ratios increased for many firms and this in turn 
impacts credit ratings. The same factors affect subcontractors also. Some prime contractors are down to 
sole source suppliers. Because the ability to use foreign suppliers is limited by ITAR and buy American 
regulations are more prevalent, low cost acquisition options are disappearing. This all tends to inhibit 
competition and increases costs. 

Aggravating the situation still further is the fact that from the mid-1980s through 2007, a number of 
major American companies have chosen to leave the defense industry - but no major non-defense firms 
have chosen to enter the industry and fill the void. Some of the reasons cited is 1) the statutory and 
regulatory constraints required by the government, and 2) the political and bureaucratic complexity of 
working with the government as both a regulator and a buyer. 
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Figure 12: - A Visual Overview of the Consolidation of US Defense & Aerospace Companies from 1993-2007 20 


20 The US Defense Industrial Base: Past , Present and Future, Barry D. Watts 
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One study correlates as much as 40 percent of DOD program cost overruns to reductions in allowable 
annual purchase quantities imposed by top-level members of the DoD/Executive branch or Congress 
(e.g. an order for "X" units was originally placed but afterwards the total order was reduced to "X-n" 
resulting in a higher per/each cost). These factors are often beyond the control of government or 
industry program managers. The net result, however, increases costs for a number of reasons including: 
1) spreading fixed costs across fewer units; 2) ordering parts that cannot be installed; 3) constraining the 
ability to buy components in large quantities at lower prices; and 4) buying more in later years after 
inflation has discounted program dollars. Moreover, cutting funding may negate contractual provisions 
and enable companies the opportunities to negotiate their way out of being charged for overruns. More 
than half the costs of a program can be “fixed costs.” For example, the cost of a design facility and the 
salaries of a design team are fixed. When the government stretches a program from two years to three 
years, such costs are extended and incurred for an additional twelve months. 

In 1983, the Air Force originally planned to procure 132 B-2 bombers; then. Defense Secretary Richard 
Cheney reduced the buy to 75 stealth bombers in 1990 and less than two years later President Bush 
ended production with a further reduced 20 aircraft purchase. In like manner, as of 2008 it appears that 
the Air Force will only take delivery of 175 operational F22 fighters whereas the number advertised to 
the industry in the late 1980s was 880 planes. 2 

The following table summarizes the burgeoning history of another DoD Program, the Evolved 
Expendable Launch Vehicle (EELV). 

Concept System development Production 

▲ ▲ A. A A A 

Program Development start and First First Initial GAO 

start production decision light-AllasV flaght-Delta IV curability review 

( 12 / 96 ) ( 6 / 96 ) ( 8 / 02 ) ( 11 / 02 ) ( 12 / 06 ) ( 1 / 06 ) 


Program Essentials 

Prime contractor: Boeing Launch 
Services, Lockheed Martin Space 

Program Performance (fiscal year 2008 dollars in millions) 

As ot Latest Percent 

10/1998 08/2007 change 

Systems 

Research and development cost 

$1,690.9 

$1,837.2 

8.9 

Program office: El Segundo. Calif. 

Procurement cost 

$14,810.2 

$30,443.9 

105.9 

Funding needed to complete: 

Total program cost 

$16,500.9 

$32,281.2 

95.6 

R&D: $0.0 million 

Program unit cost 

$91,165 

$233,922 

156.9 

Procurement: $25,156.1 million 

Total quantities 

181 

138 

-23.5 

Total funding: $25,155.1 million 
Procurement quantity: 109 

Acquisition cycle time (months) 

NA 

120 

NA 


Figure 13: - EELV Project performance" 3 


Another problem has been overestimating the transfer of learning from one unit to the next. Learning- 
curve theory, originally based on aircraft production experience during the 1930s and late 1940s, holds 
that as the number of units produced doubles, the recurring cost per unit decreases at a fixed rate or 
constant percentage. In reality, this becomes a law of diminishing returns scenario and breaks down 
when production numbers reach some level of output. However, optimistic assumptions about 
manufacturing and the notional learning curve idea of the past present an obvious temptation to low-ball 
production costs when bidding the work. Especially given the DOD recent trend to decrease number of 
units ordered. 


21 The US Defense Industrial Base; Past , Present and Future, Barry D. Watts 

22 The US Defense Industrial Base; Past , Present and Future, Barry D. Watts 

23 GAO-08-467SP Assessments of Selected Weapon Programs 
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Unsettled requirements in acquisition programs can create significant turbulence. Sixty-three 
percent of the programs we received data from had requirement changes after system development 
began. These programs encountered cost increases of 72 percent, while costs grew by 1 1 percent 
among those programs that did not change requirements. 24 



Programs 1 to 24 months late 


Programs 25 to 48 months late 


Programs more than 48 months late 


Programs on time 


Note: This reflects planned or actual delivery of initial capabilities for programs with comparable 
schedule data. 

Figure 14: - DoD Schedule Slip is 21 months on average 25 


Fiscal year 2008 dollars 



Fiscal year 



2000 portfolio 

2005 portfolio 

2007 portfolio 

Portfolio size 

Number of programs 

75 

91 

95 

Total planned commitments 

$790 Billion 

$1.5 Trillion 

$1.6 Trillion 

Commitments outstanding 

$380 Billion 

$887 Billion 

$858 Bilion 

Portfolio performance 

Change to total RDT&E costs from 
first estimate 

27 percent 

33 percent 

40 percent 

Change In total acquisition cost 
from first estimate 

6 percent 

18 percent 

26 percent 

Estimated total acquisition cost 
growth 

$42 Billion 

$202 Billion 

$295 Bill ton 

Share of programs with 25 percent 
or more increase in program 
acquisition unit cost 

37 percent 

44 percent 

44 percent 

Average schedule delay in 
delivering initial capabilities 

18 months 

17 months 

21 months 


90UC* GAO urntfm <4 DOO <*• 

Note: Data were obtained from DOD'* Selected Acquisition Reports (dated December 1 WO. 2004, 
and 2006) or. in a tow cases, data were obtained directly from program offices. Number of programs 
reflects the programs with Selected Acquisition Reports. In our analysis we have broken a tew 

Figure 15: - DoD Cost and Schedule Growth L 6 


24 GAO-08-467SP Assessments of Selected Weapon Programs 

25 GAO-08-467SP Assessments of Selected Weapon Programs 

26 GAO-08-467SP Assessments of Selected Weapon Programs 
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One major difficulty in determining project cost growth is the fact that government agencies try to put 
the best positive spin on their projects, even if it means twisting the truth. For example the GAO 
reported 27 in 2005 “The DOD does present Congress with valuable information about a program’s 
performance by comparing the latest unit cost estimate against the most recently approved baseline. 
However, this provides only one perspective on performance because re-baselining shortens the period 
of performance reported and resets the measurement of cost growth to zero. Other meaningful 
perspectives are not reported. For example, DOD reported in the 2003 SAR, the most recent available, 
that the F/A-22 Raptor program’s unit cost decreased by 0.33 percent in the previous 4 months — since 
the latest rebaselining (project rebaselined 14 times between 1992 & 2004). DOD did not report that the 
program’s unit cost had cumulatively increased by 72 percent in the last 143 months. Second, the 
change in unit cost between one budget request to Congress and the next is not measured or reported. 
For example, DOD reported in the 2003 SAR that unit cost for the Stryker program increased by 1.34 
percent in the 2 months since the latest rebaselining; it did not report that unit cost had grown by 21 
percent in the previous 12 months. The DOD reported in the December 2003 SAR that the Marine 
Corps’ H-l helicopter upgrade program’s unit cost has shrunk by 1 percent in the last 20 months; 
however, DOD did not report that the program’s unit cost had, in constant dollars, doubled in the last 87 
months.” 

Finally, the DOD has another cost avoidance technique it often uses in an attempt to limit the perception 
of cost growth. They will alter production standards and accept equipment that may not meet earlier 
specifications - then upgrade the equipment at a later date. 2 * Taken together, all of the foregoing tend to 
increase cost growth well beyond the publicly reported statistics. 

Award Fees and the DoD 

Award Fees should pay for results, not effort. Award fee contracts are not resulting in value for the 
taxpayer: One of the problems that we have in government, is that if we’re paying incentive and award 
fees, we need to pay for positive results achieved; that people do what they promise or what we need and 
what they promise, when they promised it, and at the cost that was agreed to. Unfortunately, that’s not 
the case for all too many contracting arrangements in government. They pay for effort and that’s it, not 
results. 

Despite the F/A-22 cost overruns unjustified award fees were given. On this contract, Lockheed received 
$849 million in award fees despite incurring $10.2 billion in cost overruns and delays of over two years. 
In total, Lockheed received 91% of the available award fee despite the large cost increases and lengthy 
delays. 29 

The Mercury Program 

The3 Mercury Program ran from 1961 to 1963. Its purpose was to demonstrate that humans can travel 
into space and return to earth safely. There were a total of six flights (two suborbital and four orbital). 
Alan Shepard was the first American in space (a suborbital flight) on May 5, 1961. John Glenn was the 
first American in orbit on February 20, 1962. The Mercury program cost $384 Million, almost twice the 
original estimate. 30 When President Kennedy proposed the Apollo program, the United States had 
achieved only one human space flight — Alan Shepard’s 15-minute suborbital Mercury-Redstone 
mission. 


27 GAO-05-182 Defense Acquisitions Information for Congress on Performance of Major Programs Can Be More Complete, Timely, and Accessible. 

28 Air Force wants $1.9 Billion Through FY94 to Fix Sc Improve B-1B, Aerospace Daily, May 3, 1989 

29 GAO-06-66, Defense Acquisitions; DOD Has Paid Billions in Award and Incentive Fees Regardless of Acquisition Outcomes (Dec. 2005) (GAO-06-66). 

30 A Discussion of Space Program Cost, David Novick, RAND February 1964 
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The Apollo Program 

The Apollo Lunar Program (conducted from 1962 to 1973) comprised 17 missions with six successful 
moon landings between 1969 and 1972. The total program cost (in 2005 dollars) was roughly $170 
billion. 31 That total included all research and development (R&D) costs; the expense of procuring 15 
Saturn V rockets, 16 command/service modules (C/SMs), 12 lunar modules, program support and 
management costs; construction expenses for facilities and their upgrading, and costs for flight 
operations . 

On April 25, 1958 the Air Force Ballistic Missile Division published the first development plan to 
"achieve capability to land a man on the moon and return him safely to earth." The complete 
program would be carried out in four phases: first, "Man-in-Space Soonest"; second, "Man-in- 
Space Sophisticated"; third, "Lunar Reconnaissance," exploring the moon by television camera 
and a soft landing of an instrumented package on the moon's surface; and finally, "Manned 
Lunar Landing and Return," which would first test equipment by circumlunar flights returning to 
earth with instrumented capsules containing animals. At this stage of project development, the 
payload capacity would be increased to 9000 pounds. The spacecraft would then undertake a 
full-scale flight to the moon and safe return with an animal passenger. The climax would be a 
manned lunar landing, brief surface exploration, and return. This would be followed by other 
flights to explore the lunar surface thoroughly and gather additional data. The program was 
scheduled for completion in December 1965 at a cost of $1 .5 billion. 32 

"We can give you estimates all the way to the end of the Saturn-IVB, but do you really 
want to know it?" 

Donald W Douglas - President Douglas Aircraft - 1964 


Apollo is often painted as a successful program that came 
close to its initial 1961 estimate for a 1967 moon landing. 

As can be seen from the foregoing 1958 Air Force 
estimate, the original cost was 1 .5 billion with completion 
targeted in 1965. The "actual" historical events went 
something like this. The NASA cost estimating gurus in 
1961 projected an amount close to $7 Billion to do the 
entire program. 33 34 This figure was apparently padded to 
$10-$ 12 Billion by management prior to giving that 
estimate to James Webb, the NASA Administrator. Mr. 

Webb (within hours of receiving the $10-$ 12 Billion 
figure) placed an "administrator's discount" on NASA’s 
ability to predict costs with due precision and by the stroke 
of his own pen, changed the estimate to $20 billion and 
submitted it to Vice President Linden B. Johnson. In the 
words of Robert Seamans Jr., (the Associate Administrator 
at the time) "We were aghast!" 35 This cavalier beginning 
describes how Apollo's original fiscal requirements arrived 
at the steps of the Capitol and was subsequently blessed by 
Congress. Figure 16: - Saturn V Overall Dimensions. 
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31 CBO A Budgetary Analysis of NASA’s New Vision for Space, Sept 2004 

32 Chronology of Early USAF Man-in-Space Activity, 1945-1958 (U.S. Air Force, 1965), unpublished, pp. 21-22 

33 Apollo Executives’ Meeting Proceedings Ashville, NC June 18-19 1964 

34 A Discussion of Space Program Cost, David Novick, RAND February 1964 

35 Aiming at Targets - The Autobiography of Robert C. Seamons Jr. - The NASA History Series 19%, pp. 91 
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Ironically, the $20 billion amount submitted by Mr. Webb to the Vice president appeared to be a 
completely arbitrary and highly irregular move. In anyone's book it was a radical cost estimating 
maneuver to be sure. But in the end, Mr. Webb's innate business sense and the courage to follow what 
that sense told him validated his action. It turned out to be a leadership demonstration of profound 
foresight. In the end the "real cost" of Apollo ultimately surpassed Mr. Webb's $20 billion estimate with 
a price tag of $25.4 billion as was reported to congress in 1973. The final program cost varies 
depending on what we include or exclude in the calculations, 36 37 38 but in all instances exceeds $20 
billion. 

Officially, Apollo met President Kennedy’s publicly declared goal of putting a man on the moon by the 
end of the 1960s. 39 To achieve this tremendous exploit, the original cost estimate from NASA experts 
($7 billion), was increased by nearly 3 times that amount when it became the "official" agency 
submission for the Apollo Program. 

"The fellow who proposed to do the job at the lowest estimate normally ends up by doing the 

job for twice the estimate." 

R. R. Hough - Vice President AT&T - 1964 

The greatest cost items were the Saturn V rockets @ $6.4 billion, followed by Command and Service 
Modules @ $3.7 billion. Lunar Modules @ $2.2 billion, and Manned Space Flight Operations @$1.6 
billion. 



Figure 17: - 1964 Projection of NASA Funding 


• Schedule Performance 

> Assuming 1962 official program 
start 

> Scheduled date of 1 st moon landing 
1967 

> Actual date 1 st moon landing was 
July 1969 

> 1962 -1967 = 5 years 

> 1962 -1969 = 7 years 

> (7 / 5) -1 or (5/ 2) = 40% schedule 
slip 

• Cost Performance 

> Original Estimate $7 Billion 

> Final Costs $25.4 

> (7/25.4)- 1 =263% 


36 NASA Budget http://en.wikipedia.org/wiki/NASA_Budget 

37 Ezell, NASA Historical Data Book, Vol II, 2:122-32 

38 House, Subcommittee on Manned Space Flight of the Committee on Science and Astronautics, 1974 NASA Authorization, Hearings on H.R. 4567, 93/2, Part 
2, Page 1271. 

39 CBO A Budgetary Analysis of NASA’s New Vision for Space, Sept 2004 
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The Space Shuttle 

The Space Shuttle reportedly was given initial approval with a 1 vote margin. There was much concern 
regarding estimate accuracy. Shuttle planning began in 1968 40 and the initial 1971 estimate was $10 
Billion 41 for a more complex vehicle than ultimately was built that included fly-back capability using a 
liquid booster. As a result of prevailing budget pressures the scope was cut. The 1972 final estimate 
was for the development portion of shuttle's current configuration using solid rocket boosters. The 
figure for five orbiters was near $5.5 Billion plus a 20% contingency for reserves 42 totaling 6.6 Billion. 
The Shuttle was scheduled for first horizontal flight in 1975, first vertical flight 1977, first manned flight 
by the end of 1977, and initial operating capability in 1978/1979. 43 

The January 5, 1972 estimate for the Shuttle Program called for an expenditure of $5.5 billion for 
research, development, test and engineering, together with a 20 percent overrun allowance, and $0.3 
billion price tag for launch facilities. This Shuttle would be a two-stage vehicle — consisting of a booster 
and an orbiter. Each booster would cost $50 million. At this point in time, the booster reuse capability 
was not defined. Each orbiter (which NASA claimed could be used 100 times) will cost an estimated 
$350 million. NASA held that each shuttle flight would cost less than $10 million and that the cost of 
placing a pound of payload in orbit could be reduced to less than $100. Thus, it was argued that the 
Space Shuttle would certifiably be a cost effective vehicle for the space program. 44 



NASA's 

Mathematical 

cases 


March 1972 

“Sorra 

Liquid 

• 

estimate 

booster 

booster 


(billions 

-1971 dollars) 

Expected launch vehicle costs 

$16.1 

$14.6 

$14.2 

Expected payload costs 

26.8 

26.8 

26.8 

Expected total space program 




costs 

$42.9 

$41.4 

$41.0 



Figure 19: - 1970 ’s concept for a two stage Space Shuttle 


40 William Normyle, 'Large Station May Emerge As Unwritten US Goal,' Aviation Week and Space Technology (March 10, 1969), pp 103-109. 

41 12-29-1971 NASA Letter from James Fletcher to OMB 

42 5-9-1972 NASA Letter from James Fletcher to OMB 

43 9-30-1970 NASA Letter from James Fletcher to OMB 

44 Congressional Report Cost Benefit Analysis Used in Support of the Space Shuttle Program 1972 

45 Congressional Report Cost Benefit Analysis Used in Support of the Space Shuttle Program 1972 
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Don Rice (Assistant Director, OMB) presented a developmental cost of $4 billion and that was believed 
to have come from North American Rockwell. It was subsequently discovered that Rockwell had given 
far more information to NASA than to Rice, and Rice's $4 billion left out company profit along with the 
cost of NASA program support. Indeed, it was the equivalent of numbers that NASA itself had shown. 

SPACE SHUTTLE DEVELOPMENT SCHEDULE 
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Figure 20: - 1972 1 st flight slipped 2 years 
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December 1971 planning showed that the first shuttle launch was scheduled to occur in June 1976 (or 
4.5 years from that time) with a planned launch rate of 50 to 1 50 flights per year. By the early 1 980's the 
planned flight rate was reduced to 24 flights per year. The actual 1 st launch date occurred April 12, 1981 
or 9.4 years from the December 1971 vantage point. This translated into a 100% + slip in schedule. The 
maximum shuttle flight rate ever achieved has been eight launches per year, which is % of the then 
conservative estimate of 24 flights per year. In addition, the program experienced substantial cost 
growth in achieving the "reduced" target of 8 flights per year. Had the initial flight rate had been 
insisted on; a gargantuan amount of additional money would have been needed. 



Figure 21: - Approximate Space Shuttle Budget Distribution - Important: some costs, like civil service salaries are not 
included due either to difficulty of accurately ascertaining Shuttle-relevant activity, or their insignificance. Adding these 
categories increases estimates, but probably less than 10%. Abo excluded b 3 billion in DOD support, much of which was for 
the Vandenberg CA. launch site development, and 0.5-1 billion in reimbursable costs from the DOD 46 SFCDC = Space 
Flight, Control and Data Communications. 


46 Space policy alternatives / edited by Radford Byerly, Jr. 1992 
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"In discussing Space Shuttle development approval, 1970 House Representative Joseph 
Karth (Democrat - Minnesota) was suspicious of NASA's claims for program costs. For 
projections of robotic missions, Karth referred to such projections as "asinine" and followed 
up with the words "NASA must consider members of Congress stupid idiots." 


SpaceRef.com 
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Figure 22 - 10-28-1971 Shuttle report to the administrator $ in Millions of undiscounted 
1970 dollars, assumes 514 flights. 
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Figure 23: - 12-29-71 Letter from James Fletcher to OMB 


The original concept was to use conventional rocket engines for the Shuttle's boosters. However, 
between January and March, while the development cost of a pressure-fed booster stayed virtually 
constant, the Space Shuttle Main Engine (SSME), and orbiter costs escalated sharply. A 1972 internal 
OMB memo from economist Sullivan summarized NASA's own estimate of the changes, in millions of 
dollars: 47 



1/3/1972 Est. 
(Pressure-fed) 

Pressure-fed 

Current Est. 
Pump-fed 

Solids 

Orbiter 

3,058 

3,660 

3,660 

3,750 

Main Engine 

450 

580 

580 

580 

Booster 

1,390 

1,400 

1,080 

350 

Program Support 

602 

570 

560 

470 

Total 

5,500 

6,210 

5,880 

5,150 

Cost/flight 

7.7 

9.3 

8.6 

10.4 


Figure 24: - Shuttle Booster type cost comparison in Millions 


47 SP-4221 The Space Shuttle Decision - NASA 
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Since the White House insisted that NASA stay within a $5.5 billion development cost, NASA was 
driven to the Solid Rocket Booster (SRB) option. The low estimates gave a strong case of choosing the 
solid motor even though no one had previously tried to recover and reuse a solid booster. It was decided 
that the Shuttle would use two 156-inch boosters, the largest allowable size that could be transported on 
American railroads. Only nine such solids had ever been test fired - five by Thiokol and four by 
Lockheed. It was clear that Marshall SFC would have plenty to do in bringing the SRB to a level of 
reliability that would allow delivery of astronauts to space. 

By October 1974, the Space Shuttle Main Engine's (SSME) 10 hour life had been reduced to 7.5 hours. 
Most other systems were still designed for a 10 operation life. The SRB design was delayed due to a bid 
protest by Lockheed. 48 

A 1977 NASA letter 49 to congress confirmed the current Space Shuttle design, development, test and 
evaluation (DDT&E) estimate “is still $5.22 billion in 1971 dollars”, or “$6,816 billion in the FY 1978 
budget” “after adjustment for inflation already incurred and inflation expected during the coming budget 
year.” The first manned orbital flight was projected to slip to mid 1979. Yet further in the same letter, 
seemingly in conflict with the earlier statement, it reads: "The cost of refurbishing Orbiters 101 and 102 
and procuring Orbiters 103, 104, and 105 was originally estimated at approximately $1 billion in 1971 
dollars. However, “the decision in the FY1977 budget to defer Shuttle production for one year” resulted 
in an increase to approximately $1,347 billion in 1971 dollars or $1,988 billion in 1978 dollars. Plus 
$432 million more was added for Orbiter and KSC facility ground support equipment for a two orbiter 
in flow processing capability as well as initial spares and crew equipment for the operations period. 

The actual cost through 1981's first flight ended up between $13. 6 50 and $17 51 billion (or 147% to 258% 
more than estimated). Total Space Shuttle Program cost through 1990 was approximately $65 billion, 
compared to the projected estimate of $5 1 billion. Though this seems like a relatively modest overrun 
(about 27%) it must be remembered that the $5 1 billion was to have paid for 580 flights while only 37 
flights were actually flown through 1990. The flight frequency schedule was a mere 6.4% of the 
originally projected service performance. If all 580 flights had been made as initially projected, the cost 
would have come in at 1 trillion, nineteen billion dollars - more than 35 times the estimate. Thus when 
comparing apples to apples, the optimism bias reflected in the Shuttle Program was up there with the 
worst estimates in NASA's history. 

Type of Cost Cost 


Design, Development, Testing and Engineering 13 , 138 

Production 3,948 

Construction of Facilities 703 

Total 17,789 


Figure 25: - Shuttle System Non-Operational Expenditures FY 1972-1982 in Millions of 
1982 Dollars. 52 


48 Flight International, NASA’s Budget gets a Boost, 10-3-1974 

49 4-12-1977 NASA Letter from James Fletcher to Subcommittee on Space Science and Applications Committee on Science and Technology 

50 Current $ - Pielke 1994, Gehman 2003, 2004 Economic Report of the President 

51 Aviation Week, Space Shuttle Value Open to Interpretation 7-26-1993, Period from 1971-1982 

52 CBO Pricing Options for Space Shuttle 1985 
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As shocking as the cost growth in the shuttle program was, it was not entirely unanticipated. The 
President's Science Advisory Committee created a special panel to examine the issue, and appointed 
Alexander Flax as chairman. An October 19, 1971 interim report 53 stated: 

“Considering all of the technological and operational unknowns involved 
in the shuttle development and the fact that no vehicles of similar function 
have ever been designed before or have ever operated over the range of 
flight regimes required for the shuttle, prudent extrapolation of prior 
experience would indicate that estimated development costs may be 30 
to 50 percent on the low side. Thus, the estimates of $6.5 billion in 
RDT&E for the Mk I & Mk II shuttle program may range between $8.5 to 
$10 billion, reflecting increased program costs of $2.5 to $3.5 billion. 

Similar uncertainties must be considered to apply to other non-recurring 
costs such as production and facilities (amounting to about $4 billion). 

Thus a possible cost uncertainty of about $5 billion for total program costs 
might be envisioned giving a high estimate of_ total non-recurring cost of 
about $15 billion. ” 

“The operating cost estimates of $5.5 million per flight for the shuttle, 
within narrow limits, must be considered to be a very rough estimate at 
this time, particularly for the early years of shuttle operation. The actual 
value will depend upon the time between overhaul of equipment not yet 
designed, refurbishability of thermal protection system materials not yet 
out of the laboratory, and on the feasibility of operating the shuttle in an 
"airline" mode radically different from all past experience in space 
operations.” 

The President's Science Advisory Committee was on the right track. Their main oversight was simply 
not going nearly far enough in suggesting additional dollar requirements. It is generally acknowledged 
that the Shuttle Program has not met its original objectives either in frequency of flight or efficiency of 
operation. This is not to say that the Shuttle is not a grand technical achievement — it is in fact an 
engineering marvel! However, the specifics of the performance shortfall against the advertised 
projections and estimates are as follows 54 . 

• The number of flights have been 1/1 5th the 1 972 prediction. 

• 5 Operational Shuttles were budgeted in 1971 but only 4 were built. The "Enterprise" which 
was intended to be the 5th orbiter, was used for flight tests but was constructed without engines 
or a functional heat shield. It was not retrofitted for orbital flight due to major design changes. 

• The average cost per flight was over 1 9 times what was promised. 

• If the Shuttle were to fly 8 flights per year (which is the maximum ever launched in one year), at 
$4 billion in average annual appropriations, it would take approximately 68 additional years and 
approximately $270 billion to reach the original goal of 580 flights. 

• Original Space Shuttle requirement called for 65,000 lbs to a 180 km, 28 degree orbit. 

• Average capability of the three orbiters is -49,000 lbs to the same orbit (a decrease of -25%). 


53 SP-4221 The Space Shuttle Decision - NASA 

54 Space policy alternatives / edited by Radford Byerly, Jr. 1992 
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Fiscal Year 


Figure 26: - Historical Shuttle Launch Rate per year - average rate is 4.4 flights per year. 

Up to February 2005, the entire Shuttle Program cost was approximately $145 billion with $112 billion 
of that amount occurring since it became operational in 1981. Furthermore, the average cost per flight 
has been about $1.3 billion over the life of the program with about $750 million per flight over its five 
most recent years of operation. 55 

In summary, the Space Shuttle is a testament to the ingenuity and flexibility of NASA engineering - 
leaving a legacy of accomplishment that even the most cynical can admire, but the business case and 
financial metrics history connected with those technical achievements has fallen far beneath the super- 
optimistic estimates put forth when selling the concept to congress and the public. 

The International Space Station 

The Space Station story is an internationally encumbered and complex one - exceeding in sophistication 
the already difficult "normal" cost estimating/schedule setting arena. In his January 25, 1984 State of the 
Union address, President Reagan directed NASA to build a space station within a decade and to invite 
other countries to join the endeavor. In 1991, the US House of Representatives held the first of 22 
separate votes on whether to proceed with the program. It came within 1 vote of cancellation in 1993. 
Construction began in 1998. 



Figure 27: - The International Space Station Baseline Configuration 


» MSNBC, Space Shuttle Cost Get a Reality Check 2-11-2005 
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"For years, the agencies (NASA's) cost overruns and schedule delays supplied 
ammunition for annual congressional attempts to kill the International Space Station." 

Florida Today 


Comparing costs associated with the space station is extremely difficult since over the years the scope 
has changed significantly. The initial $8 billion estimate, expressed in FY1984 dollars (as if all funds 
were paid out that year). The current design estimate has ballooned to $45 Billion (FY 1985-2005, in 
current dollars), which reflect current and prior year spending unadjusted for inflation, plus future year 
spending that includes a factor for expected inflation. Cost growth and schedule delays over the past 21 
years have subjected space station to repeated downsizings and consequent reductions in capabilities 
from 1984 design. Space station was originally presented to Congress as a facility with eight functions. 
Within five years, it had been reduced to one function — a laboratory for world class research. Full 
assembly was originally intended for completion by 1994. Completion now scheduled for 2010. 


Cost Estimates For U.S. Portion of the International Space Station: 1984-2005“ 

Year 

Estimate* 

1984 

$8 billion (FY1984 dollars, R&D only, no shuttle launches). 

April 1987 

$16 billion, following restructuring in which program was split into two "phases" $12.2 
billion for Phase 1; $3.8 billion for Phase II (FY1984 dollars, R&D only, no shuttle 
launches). 

April 1989 

$30 billion for Phase 1 (real year dollars (RYD),* through assembly complete, 
including shuttle launches during assembly and other costs-such as the Flight 
Telerobotic Servicer and ground facilities). Phase II was "indefinitely postponed," so 
it is not included in this or subsequent cost estimates. 

Early 1990 

$37 billion (RYD, through assembly complete, including shuttle launches during 
assembly and other costs). 

December 1990 

$38.3 billion (RYD, through assembly complete, including shuttle launches during 
assembly and other costs). 

March 1991 

$30 billion (RYD, through permanent human capability, including shuttle launches 
during assembly and other costs). 

Nov. 1993 

$17.4 billion. This adjusted cost followed termination of Space Station Freedom 
(which was to be solely a US constructed space facility) and introduction of the 
International Space Station (which was a planned partnership with Russia and other 
nations). (RYD, development costs through assembly complete, no shuttle 
launches, includes costs for science experiments). 

July 1996 57 

GAO $17.4 billion to develop and operate the ISS; from October 1993 through 
completion of assembly in space, currently scheduled June 2002. With $6.3 billion 
for prime contractor's costs, $5.5 billion for development of ground-based and on- 
orbit capabilities, $2.6 billion for development of on-orbit research facilities & other 
research, $3 billion for financial reserves; 

NASA did not budget the full amount of contract estimates because it believed that it 
can negotiate authorized contract changes; 

As of April 1996, the prime contractor was $89 million over budget and $88 million 
behind schedule; there are several instances where the prime contractor's total cost 
estimates neglect to recognize over-budget conditions; 

March 1998 

$21.3 billion (RYD, development costs through assembly complete, no shuttle 
launches, includes costs for science experiments). 


^CRS NASA'S Space Station program Evolution and Current Status 4/4/2001 
57 G AO/T -NSI AD-96-2 1 0 
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April 1998 

$24.7 billion (not a NASA estimate), but was an independent "Cost Assessment and 
Validation Team" estimate headed by Jay Chabrow which concluded what the cost 
would be through assembly complete. 

June 1998 

$22.7 billion (RYD, development costs through assembly complete, no shuttle 
launches, includes costs for science experiments). NASA did not accept the 
Chabrow figure, but agreed program would cost $1.4 billion more. 

June 1998 58 

GAO $95.6 billion total space station costs • Regarding shuttle support, our 1995 
estimate was based on 35 flights during development and 50 during operations. 
However, NASA’s 1998 estimate was based on 43 flights during development, 
including 2 additional flights to the Russian space station Mir, 1 flight to test the crew 
return vehicle, and flights required by adoption of Revision C to assembly sequence. 
NASA continues to estimate that 50 flights will be needed during operations. 

June 1998 s * 

GAO $21.9 billion development cost - higher costs $21.9 versus $17.4 billion — 
attributable to schedule delays, additional prime contractor effort not covered by 
funding reserves, additional crew return vehicle costs, & costs incurred as a result of 
delays in Russian-made Service Module. 

Increased in-house personnel costs during development — $2.2 billion versus $0.9 
billion — attributable to longer development program, higher estimated personnel 
levels, and more inclusive estimating methodology. 

February 1999 

$23.4-26 billion (RYD, development costs through assembly complete, no shuttle 
launches, includes costs for science experiments). 

February 2000 

$24.1 -$26.4 billion (RYD, development costs through assembly complete, no shuttle 
launches, includes costs for science experiments). 

March 2001 
(Under 
Discussion) 

$22-23 billion, assuming termination of construction after completion of "U.S. Core" 
and attachment of European and Japanese lab modules (RYD, development costs 
through completion of the "U.S. Core”; no shuttle launches; includes costs for 
science experiments, reduced 40% from previous estimates). 

January 2001 60 

$30 Billion NASA explained that program managers had underestimated the 
complexity of building and operating the station.- Scope was cut in Feb 2001 to stay 
under $25 Billion cap imposed by congress. 

October 2003 61 

GAO $32 billion appropriated by congress since fiscal year 1985 for program. 

To date, NASA has not fully estimated the potential increased costs and future 
budget impact incurred due to the grounding of the space shuttle fleet. However, it 
has identified a number of factors that will likely result in increased costs — including 
the need to extend contracts to complete development and assembly of the station. 

NASA estimates impact to station program from Columbia accident to be $22 million 
in fy 2003 and up to $72 million in fy 2004. NASA maintains that an assessment of 
total impact cannot be accomplished prior to the fy 2006 budget submission in 
February 2005. 

September 2004 

$32,878 Billion'* 

April 2005 63 

$45 billion - $35 billion appropriated by Congress for program (FY1 985-2005 
concurrent $) and NASA estimates it will cost another $10 billion through the end of 
construction in FY2010. (Estimates do not include shuttle launch costs.) 

Cost growth and schedule delays over the past 21 years have subjected space 
station to repeated downsizings & consequent reductions in its capabilities. 


Table 1: - NASA ’s Cost Estimates For U.S. Portion of the International Space Station: 1984-2005 


58 GAO T-NSIAD-98-212 SPACE STATION U.S. Lifc-Cydc Funding Requirements 

59 GAO T-NSIAD-98-212 SPACE STATION U.S. Life-Cycle Funding Requirements 

60 CRS The International Space Station and the Space Shuttle 11-2008 

61 GAO 04-201 T Shuttle Fleet’s Safe Return to Flight Is Key to Space Station Progress 10/29/03 

62 CBO A Budgetary Analysis of NASA’s New Vision for Space Exploration September 2004 

63 CRS NASA’S Space Station program Evolution of Its Rationale and Expected uses 4/30/05 
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The above table is particularly revealing as to the volatility of both the aggregate costs and the sluggish 
timetable for the International Space Station. Of all the programs cited in this paper. The 1ST provides a 
maze of fluctuating cost and schedule estimates as the years have unfolded. After evaluating the above 
table, the words of Robert Young from 1964 take on greater meaning. 

"The time has GONE when we can re-plan and re-estimate every year." 

Robert Young - NASA - 1964 

The fiscal illness of failing to identify the entire cost scenario early in the business process has been with 
NASA a long time. The jury is no longer out on this fact. The agency’s cost estimating community 
could not reasonably require more proof than now exists. The question now is: how does NASA go 
about altering its "below-the-target" precedent and fixing the cost/schedule realism problem with that 
same professional skill which has always characterized our efforts in finding technical solutions to 
difficult engineering challenges? An answer to that question is introduced at the end of this paper. 

Quotes Worth Remembering from the Apollo Era 

Having completed a cursory review to this point of NASA's most prominent and visible programs, a 
series of quotes follow from people of influence during the Apollo Program era 64 . Though these 
statements were made nearly half a century ago, they are still timely and appropriate in our current 
business climate. Our object in citing these long ago statements is to drive home the point that among 
the most persistent realities in the science and engineering construction world, none is so well validated 
as the precedent of underestimating cost and over-optimizine schedules . 

"One of the things that is essential is a realistic set of assumptions on which your 
estimate is based. If conditions change, you modify the assumptions, and you modify the 
cost." 

Charles Beck - President, Philco - 1964 


"One of the characteristics (that is) different about the Apollo program - and the 
preceding programs is the fact that initially people (Congress) were willing to fund almost 
any level of money, and they literally did. There wasn't a ceiling established when the 
program was first formulated. The problem is when the first round of contracts was 
negotiated, and the first round of run-out costs developed, the estimates were $7 billion. 
The first statement to Congress was that the cost would be between $30 and $40 billion." 

Dr. George Mueller - NASA - 1964 


"There has been, in all R&D programs, a tendency to underestimate the cost. In DOD 
programs, where there’s been a lot of very careful attention given to costs in the past two 
or three years, history shows the actual cost to be on average 3.2 times the initial 
estimated costs." 


Maj. Gen. Samuel C. Phillips - 1964 


64 Apollo Executives Meeting Proceedings, June 18,19 1964 
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"I can think of a lot of programs in the Boeing Company where, if the estimate had been 
realistic, you wouldn't have had the program. And that is the truth." 

W. M. Allen - President, Boeing - 1964 


"As the advocates of new programs, government agencies have often encouraged 
contractors to estimate costs optimistically, recognizing that headquarters might be 
shocked out of supporting a program where true costs were revealed at the outset. They 
have sought to disclose cost increases gradually, after programs have gained momentum 
and cancellation has become difficult." 


Dr. Frederic Scherer - Harvard - 1964 


"We have practically a whole generation of engineers who have never had to live within 
estimates they have made. They have lived in an atmosphere where lots of things were 
there to hang your hat on for overruns, and there's a lot of tongue-in-cheek estimating. 
You will find engineers, once they get there, (and I have had some sad experiences on 
that) will start thinking a little better." 

R.R. Hough - Vice President, AT&T- 1964 


"You have basically built into the entire system a series of events which very much leads 
to under-estimating what the program is going to cost." 

J. S. Parker - Vice President, GE - 1964 


"We had a program which we estimated at $115 million - it ended up costing $427 million. 
The program was overrun by a factor of three. The initial estimates were never furnished 
to the people who had to get the money." 

J. S. Parker - Vice President, GE - 1964 


Mars Science Laboratory 

The Mars Science Laboratory (MSL), which has ballooned to a $2.3 billion price tag, is a good example 
of NASA's estimating approach. In 2003, the cost for the MSL was put at $650 million on the National 
Academy of Sciences wish list. 65 Doug McCuistion (who heads NASA's Mars exploration program) 
said the $650 million estimate was not an official NASA projection, it was put together by a panel of 
planetary scientists in 2003 66 , and the proper estimate (to start) should have been $1.4 billion officially 
approved at PDR. 

However, what McCuistion doesn’t say is that the $650 million estimate WAS used to initially approve 
the project. By December, the number was up to $1.9 billion. Then technical problems delayed launch 
plans from 2009 to October 201 1 adding another $400 million in costs, including cost of Mars 
operations from 2012 to 2014. The extra money came from cuts to other science projects. 


65 NASA’s Cost Overruns Soar, Too, Baltimore Sun, 3-9-09 

66 Mars Science Laboratory: the budgetary reasons behind its delay. The Space Review 3-2-09 
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'The costs of badly run NASA projects are paid for with cutbacks or delays in NASA 
projects that didn't go over budget. Hence the guilty are rewarded and the innocent are 
punished." 

Allen Stem - Former Chief of NASA Science Mission Directorate 


MSL has been touted as the most capable scientific mission every sent to another planet. Most 
instruments are unique, cutting-edge items never used before on a space mission. It is also large, at 
twice the size and five to six times the weight of a Mars Exploration Rover (MER). It requires new 
entry, descent, and landing (EDL) technology. MER pushed the limits of parachute landing, and MSL 
would be the first to try a revolutionary lifting body and “sky crane” that would maneuver the 
envisioned 900-kilogram rover downward to within feet of the surface, and drop it ever so lightly to the 
ground. MSL was built from the ground up to have a fully redundant Command and Data Handling 
System (CDHS) unlike the MER rovers which had non-redundant “single string” systems. 


Industry Data 

Other industry cost/schedule comparisons which while dissimilar to NASA's development projects are 
useful for trend analysis. 

The conclusions of a study titled "Megaprojects and Risk" 67 which examined data from 258 large 
transportation projects in twenty nations around the world over the last 70 years, appears to correlate 
strongly with NASA’s history of project cost growth. Its findings were that: 

• Cost underestimation and overrun have not decreased over the past seventy years. No 
learning seems to take place; 

> There have been a few studies that suggest cost growth for NASA and DOD has 
decreased in later years. However, both the number and size of projects 
available for analysis diminished during this period. This may be a factor. 

• It is found with overwhelming statistical significance that cost underestimation and 
overrun cannot be explained by error and seems to be best explained by strategic 
misrepresentation, namely lying, with a view to getting projects started. 

• Cost overruns of 50 per cent to 1 00 per cent in real terms are common in megaprojects; 
and overruns above 100 per cent are not uncommon. 

• Actual project performance/viability typically does not correspond with forecast viability, 
the latter often being brazenly over-optimistic. 

Cost estimates at each successive stage of progress typically move toward a smaller number of options, 
greater detail of designs, greater accuracy of quantities, and better information about unit price. Thus, 
cost estimates become more accurate over time, while the cost estimate at the time of making the 
decision to build is far from final and often misses the mark by a large error of magnitude. Existing 
research indicates that project promoters routinely ignore, hide, or otherwise leave out important project 
costs and risks in order to make total costs appear low or within a pre-bid cost/schedule ceiling level. 

Costs are underestimated in almost 9 out of 10 projects. For a randomly selected project, the likelihood 
of actual costs being larger than estimated is 86%. The likelihood of actual costs being lower than, or 
equal to, estimated costs is 14%. 

• Actual costs are on average 28% higher than estimated costs (sd=39). 

67 Megaprojects and Risk: An anatomy of Ambition, by Bent Flyvbjerg, Nils Bruzelius, and Werner Rothengatter (Cambridge University Press, 2003). 
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• We reject with overwhelming significance the thesis that the error of overestimating 
costs is as common as the error of underestimating costs (p<0.001 ; two-sided test, using 
the binomial distribution). Estimated costs are statistically biased, and the bias is caused 
by systematic underestimation. 


Project Type 

Number of 
Cases 

Average Cost 
Escalation 
(%) 

Standard 

Deviation 

Level of 
Significance 
(p) 

Rail 

58 

44.7 

38.4 

<0.001 

Fixed-Link 

33 

33.8 

62.4 

<0.004 

Road 

167 

20.4 

29.9 

<0.001 

All Projects 

258 

27.6 

38.7 

<0.001 


Figure 28 - Inaccuracy of transportation project cost estimates by type of project 68 


When Eurotunnel (the private company that owns the tunnel [referred to as Chunnel] under the English 
Channel) went public in 1987 to raise funds for the project, investors were told that building the tunnel 
would be relatively straightforward. Regarding risks of cost escalation, the prospectus read: 


Whilst the undertaking of a tunneling project of this nature necessarily involves certain 

construction risks, the techniques to be used are well proven The Directors, having 

consulted the Maitre d’Oeuvre, (project manager) believe that 10% (contingency). . . 
would be a reasonable allowance for the possible impact of unforeseen circumstances 
on construction costs. 

(The Economist “ Under Water, ” October 1989, p. 37-38 j 


The Chunnel project, which opened May 6, 1994, subsequently ended up being 80 percent over on 
construction costs and 140 percent over on financing costs. Consequently, the project did not make any 
profits until 2007 and didn’t pay its first dividend (four European cents per share) until 2009 69 . 

Similarly, a 1988 RAND 70 study found that for 52 very large civilian projects with an average cost of 
about $2 billion (the smallest was 500 million in 1984 dollars), the average cost growth was 88 percent 
(measured from the beginning of the detailed engineering phase). 


Facility Type Mean Std. Dev Minimum Maximum N 


Refineries 

1.63 

0.52 

0.99 

2.54 

12 

Process Plants 

1.67 

0.68 

0.98 

3.22 

16 

Minerals Extraction 

1.99 

0.86 

1.27 

3.71 

7 

Civil/Transport 

2.14 

1.26 

0.97 

4.53 

6 

Nuclear Plants 

2.57 

0.67 

1.63 

3.41 

6 

All Projects 

1.88 

0.80 

0.97 

4.53 

47 


Figure 29: - Cost growth in the RAND Megaprojects database 


68 Underestimating Costs in Public Works Projects Error or Lie? APA Journal, Summer 2002 Vol. 68, No. 3 

69 Eurotunnel Will Pay its First Dividend, WSJ 3-5-09 

70 RAND Understanding the Outcomes of Megaprojects - 1988 
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Facility Type Mean Std. Dev Minimum Maximum N 


Refineries 

1.08 

0.16 

0.91 

1.50 

12 

Process Plants 

1.16 

0.18 

0.82 

1.45 

16 

Minerals Extraction 

1.25 

0.29 

0.93 

1.63 

7 

Civil/Transport 

1.12 

0.28 

0.97 

1.80 

6 

Nuclear Plants 

1.39 

0.20 

1.19 

1.60 

6 

All Projects 

1.17 

0.23 

0.82 

1.80 

47 


Figure 30: - Schedule slippage in the RAND Megaprojects database 

RAND attributed this cost growth and schedule slippage to: 

• Project complexity 

• Poor project definition 

• Changes in project scope 

• Regulatory issues 

• Degree of technical innovation 

• Type of ownership (e.g., public, private, or joint) 

• Extent of existing infrastructure 

• Faulty execution 

Several spectacular examples of cost overrun and schedule slippage are 71 ; 

• Suez Canal (1 ,900 percent) 

• Sydney, Australia Opera House (1 ,400 percent) 

> The original 1957 estimate was $7 million with a completion date of 1963 72 . 

> The final cost came in at $102 million with a completion date of 1973. 



Figure 31: - The Sydney Australia Opera House 


• Concorde - the supersonic airplane (1,100 percent) 

• Boston's Big Dig experienced a 275 percent overrun and a total cost of $1 1 billion. 


71 http://www.answcrs.com/ topic/ cost-ovcmm-3 

72 http://www.answcrs.com/topic/ sydney-opcra-housc 
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The Transportation Security Administration (TSA) 

In February 2002, the Transportation Security Administration (TSA) awarded a $104 million contract to 
NCS Pearson, Inc. to test and hire airport passenger and baggage screeners. In less than one year, the 
contract ballooned to $741 million. 73 Despite this expenditure, the rate at which screeners fail to detect 
weapons has remained unchanged for over four years. 74 

In August 2002, TSA entered into a $1 billion contract with Unisys Corp. to upgrade airport computer 
networks. 75 It appears that Administration officials misled Congress about the true costs of the contract. 
According to the IG, contract officials at TSA estimated that the contract costs would reach $3 to $5 
billion but decided to set an artificial ceiling of $1 billion in the public forum. The former chief 
information officer at TSA said that he was instructed by senior administration officials to cite the $1 
billion cost figure to congressional officials, which was “a number out of the air” that “would be more 
palatable.” 76 


Project Scope 


Research by the Construction Industry Institute (CII) shows project scope is the major determinant of the 
cost of a project. 77 CII research also shows that the lack of scope definition is the root cause of cost 
overruns, late completion dates, excessive rework, unnecessary disputes among project participants, and 
other problems associated with construction projects. 78 

Cost Overrun in Major Construction Projects Completed Between 1956 and 1977 

Adjusted for Unanticipated Inflation and Changes in Project Scope 

hitial Estimate Actual Result Ratio After Adjusted Compound Annual 

Rate of Cost 







Unadjusted of 

Fa 

For Change 

(Xerruns, after 


Amount 


Amount 

Date 

Ratio of Final to 

Unanticipated in Scope of 

Adjustments in 

FVoject 

(mifions) 

Date 

(rniions) 

Completed 

Initial Cost 

Inflation 

R-oject 

Rarcenta 

Bay Area Rapid Transit Authority 

996.0 

1962 1640.0 

May-76 

1.647 

1 297 

1.037 

0.31 

New Orleans Superdome 

46.0 

1967 

178.0 

Jul-75 

3.870 

3.219 

3.219 

15.73 

Toledo Edison’s Davis-Besse Nuclear Power Plant, Ohio 

305.7 

1971 

466.0 

May-75 

1.524 

1.401 

1.401 

11.89 

Trans-Alaska Oil Pipeline 

900.0 b 

1970 7700.0 c 

Jul-77 

8.556 c 

6.926 

4.250 

22.96 

Cooper Nuclear Station, Nebraska Public Power Dist 

184.0 

1966 

395.3 

1974 

2.148 

1.748 

1.748 

7.23 

Rancho Seco Nuclear Unit No. 1, Sacramento 

142.5 

1967 

347.0 

\ 1974 

2.435 

2.026 

1.239 

3.11 

Dulles Airport, Washington, D.C. 

66.0 c 

1959 

108.3 c 

1,962 

1.641 c 

1.641 d 

1.486 

14.10 

Second Chesapeake Bay Bridge 

96.6 c 

1968 

120.1 c 

Jun-73 

1.243 c 

1.104 

1.104 

2.00 

Frying Pan Arkansas Project 









Ruedi Dam 

12.8 c 

1962 

22.9 

1972 

1.789 c 

1.636 

1.145 

1.36 

Sugar Loaf 

6.1 

1962 

10.2 

1973 

1.672 c 

1.500 

1.500 

3.75 

Boustead Tunnel 

9.2 c 

1962 

21.2 c 

1973 

2.304 c 

2.078 

1.233 

1.92 

Rayburn Office Building, Washington DC 

64.0 c 

1956 

98.0 c 

Jun-66 

1.531 c 

1.531 d 

1.342 

2.99 

Weighted Average 





3.93 

3.21 

2.21 

10.07 


SOURCE Walter J. Mead, With George W. Rogers, and Rugus Z. Smith. Transporting Natural Ges fromthe Anarbc, American Enterpirse hstitute for ftjbiic Weh Research, 
Washington. D.C, 1997. pp 88-89 

a The compound annual rate expression is only used as a convenient method of comparing initial cost estimates w th the sum of al actual costs at the termination of the project. 
This device permits a compare on of overruns on several projects construction periods. 

b ki ktey 1974, the Alyeska P peine Service Co. re-estimated capital cost at $4 mlion, then in October 1974. costs were again estimated at $6 txfton for the completed pipeline. By 
June 1975, the estimete was rased to $6,375 biSon. h 1989, the $900 mton (cost estimate fa Alyeska assumed a capacity of 500 nrtVd. The scope was changed to permit a 
capcfy of 12 mko b/d. The cost of this change in scope w as $700 rh*on. rasing the initial capital cost estimate to $1.6 bffon 

c Does not include interest 
d Observed inflation less than anticipated. 

Figure 32: - RAND Review of Cost Estimating for New Technology 


73 Letter from Peter A. Iovino, Assistant Administrator for Legislative Affairs, Department of Homeland Security, to Rep. Henry A. Waxman (Sept. 2, 2005) 

74 Contracting Rush for Security Led to Waste, Abuse, Washington Post (May 22, 2005). 

75 Department of Homeland Security Inspector General, Transportation Security Administration’s Information Technology Managed Services Contract (Feb. 
2006) (OIG-06-23). 

76 Contractor Accused of Overbilling U.S., Washington Post (Oct. 23, 2005). 

77 Construction Industry Institute, Project Change Management, Special Publication 43-1 (Austin, Tex.: November 1994). 

78 Construction Industry Institute, ImprovingEarty Estimates , Implementation Resource 131-2 (Austin, Tex.: September 1998). 
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Pharmaceutical Development Projects 

A further illustration of our findings regarding the ever-common occurrence of expanding development 
costs and time, consider the results of an early study in the drug industry. 79 Detailed data was obtained 
for two major drug firms regarding errors in the cost and schedule estimates that were made at the 
beginning of chug-development projects. Over 80 percent of the projects in one firm exceeded actual 
costs and schedules. The average ratio of actual to estimated cost was 1.78 to 1.0. The average ratio of 
actual to estimated time was 1.61 to 1.0. Cost and time estimates proved less reliable for new chemical 
entities than for compounded products and alternate dosage forms. In the development of proprietaiy 
drugs, the average ratio of actual to estimated cost was 2.11 to 1 .0, and the average ratio of actual to 
estimated schedule was 2.95 to 1.0. Again, the overruns became even greater for the more ambitious 
projects. 

The Congressional Visitors Center Project 

Though Congress throws tantrums when government agencies overrun projects, they haven't performed 
any better themselves. Thq numbers vary depending on the source, but the Congressional Visitors 
Center (CVS) is commonly acknowledged as a boondoggle 80 . 

The 580,000 sq. ft. subterranean facility was initially conceived in the mid 1970’s 81 and in the early 
1990s was projected to cost $71 million. 82 Construction began in June 2000 with a budget of $265 
million and a completion date of 2004. 83 Construction was finally completed five years behind schedule 
in December 2008 for a whopping $621 million or a 775% cost growth above the initial budget estimate 
and with a 234% cost growth against the construction budget. 



Figure 33: - Inside the Congressional Visitors Center 


79 NIST Estimating Social and Private Returns from Innovations Based on the Advanced Technology Program: Problems and Opportunities 

80 Gateway to American History or Fort Capitol? Examining the Policy Evolution and Project Management of the U.S. Capitol Visitor Center Jocelyn Jones 
Evans Angela Achen 

81 CRS Report for Congress The Capitol Visitors’ Center: An Overview Stephen W. Stathis 

82 http://biz.yahoo.com/bw/081 202/2008 1 202005386.html?.v= 1 

83 http://www.time.com/time/nation/article/ 0,8599,1 625694,00.html 
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European and Australian Perspectives 

A number of European and Australian studies have demonstrated that initial estimate optimism is so 
pervasive, that the English government recommends project appraisers should make explicit “uplift” 
adjustments to the estimates of project costs, benefits, and duration based on empirical data to inform 
project decisions 84 . 

The English recommend the following adjustments for cost and schedule, always starting with the upper 
bound, and subjectively adjusting, based on project perceived risk, project phase, in place mitigation. 
The selected optimism bias adjustment factor is then multiplied by the projects total cost and schedule 
respectively. Clear and tangible evidence of risk mitigation must be observed prior to reducing bias 
optimism adjustment. 


Optimism Bias (%)* 


Project Type 

Works 

Duration 

Capital 

Expenditure 

Upper 

Lower 

Upper 

Lower 

Standard Buildings 

4 

1 

24 

2 

Non-standard Buildings 

39 

2 

51 

4 

Standard Civil Engineering 

20 

1 

44 

3 

Non-standard Civil 
Engineering 

25 

3 

66 

6 

Equipment/Development 

54 

10 

200 

10 

Outsourcing 

N/A 

N/A 

41* 

0* 


85 

Figure 34: - English Optimism Bias Adjustment - Project appraisers (analysts) should note 
that the upper bound percentages relate to the average historic optimism bias found at the 
outline business case stage for traditionally procured projects. Higher optimism bias 
adjustments may therefore be required at an earlier stage in the appraisal process 


A Closer Look at the JCL Problem 

The current confidence level assessment approach used by NASA has several critical deficiencies. 
There are key factors excluded from our estimates approach that ought to be reflected, e.g. risks that are 
highly likely to occur over the life of the program. 

Project costs often exceed the stated maximum probable cost (this is a result of multiple things that have 
been excluded). For example the Ground Operations (GO) International Space Station, Initial 
Operations Capability (ISS IOC or LEO IOC) budget increased substantially between 2007 & 2008 as a 
result of new content. In theory, we should be at ~85-90% confidence level. However, since the 
potential of new content was excluded from our analysis we are not. Congress will not care that the 
scope changed due to "new requirements" which in fact are not really new but were simply not included 
in the original project baseline "back when." The root requirement remains the same as it was when we 
started - e.g. to develop and use available capability to put a man on the moon and beyond. 


M The Green Book - Treasury Guidance 2003 

85 The Green Book Supplementary Guidance - Optimism Bias 2003 
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Total Mission Cost (FY07SM) 

86 

Figure 35: - David Bearden Chart 

Another question that crops up when formulating a cost risk policy is that of determining which risks to 
consider as part of a complete analysis. In terms of policy, a desired characteristic of any risk analysis is 
that it be comprehensive, reflecting all relevant, identifiable uncertainties. If an analysis is not truly 
comprehensive, factors that can drive the overall cost uncertainty will likely be omitted. Any resulting 
analysis will then have under-specified the true cost. Although it is impossible to have a detailed 
comprehensive list of risks for all programs, it is possible to identify broad areas of risk and uncertainty. 
Risks that are common to programs should always be considered. True risk distributions may really be 
bimodal, trimodal or even multimodal depending on the sequence of events, and decisions made by the 
project manager and the whims of congress. In other words, decisions that are made along the path of 
programs execution, can introduce alternative risk distribution curves in how program really unfolds. 


"By their very nature, we have limited experience regarding rare events, and a purely 
statistical determination is not feasible" 

RoSrinivasa Varadhan - Mathematician 


The cost and schedule growth chart below from PA&E is for semi routine projects currently in 
development. 


«j, mi .Cost & Schedule Growth from Initial(SRR) to Current Estimate 
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Figure 36: - Cost and Schedule Growth for 25 Robotics Missions in development 12/2007 PA&E 


86 Perspectives on NASA Mission Cost and Schedule Performance Trends — GSFC Symposium, Aerospace David Bearden 2008 
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SRR to 11 -5-07 /Cost 


0.00 239 



Figure 37: - Cost Growth for 25 Robotics Missions in development 12/2007 PA&E 


Here are the best fit distributions showing cost and schedule growth of the semi routine robotic missions 
shown above. Most of these missions are still in development. Realistic JCL estimate ranges must meet 
or exceed those reflected in these semi-routine missions. If not we are probably not being realistic. 



Figure 38: - Schedule Growth for 25 Robotics Missions in development 12/2007 PA&E 


The PERT Approach 


A Well Known 1960s/ 1970s Attempt 


DOD and NASA GUIDE 


PERT, an acronym for Program Evaluation and Review Technique was 
the original probabilistic scheduling method and was later expanded to 
include cost. Rumored to have emanated from West Germany 
sometime in the 1950s, PERT was selected in late 1957 as a method to 
plan and accelerate the Navy’s complex $11 billon (1967 $) Polaris 
missile program. A RAND report 87 suggests that PERT was only used 
on a small portion of the Polaris program. PERT relied on the use of 
subject matter experts to determine the optimistic, probable, and 
pessimistic time estimates. The possibility of natural disasters or any 
other unusual events were NOT considered in the analysis. In the early 
1960s PERT results & techniques were tied to cost, and adopted by the 


87 RAND Quantitative Risk Analysis for Project Management, 2004 


PERT COST 
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project management community. Its primary goal was to deflect congressional interference by 
providing a cover of disciplined, quantitative management oversight carried out by modem 
methodologies. A 1960’s Air Force study concluded that PERT costs averaged between 0.1%-0.5% of 
total project costs. 

PERT’s reputation rose to grandiose proportions. PERT was praised by Project Management Institute 
(PMI) and during the late 1960s and early ‘70s. PERT was applied to just about everything around the 
world by many organizations including NASA Soon thereafter it fell out of favor, so that by mid to late 
1 970s (despite the hype) users concluded that PERT did not work very well on most projects. PERT has 
been subsequently referred to as a “gimmick” by a Polaris program expert. 88 Meant to dazzle and 
inspire congressional confidence. In fact, a Polaris follow-on project (the Trident program) using PERT, 
earned the unsavory reputation of having been the single worst managed project in US military 
history 89 . 

It has been theorized that part of the problem with PERT was the shortage of computer processing 
power. However, when the PC and its processing capacities were exploding in the late ‘70s and early 
‘80s, PERT disappearance was unabated. If PERT were truly value added, it would have experienced a 
resurgence, which did not happen. This suggests other reasons for its demise. From our analysis, 
PERT'S shortcomings included the predominant problems of 1) optimism bias and 2) the fatal 
assumption that there is no correlation between project activities and the exclusion of negative impacts 
due to unlikely events. 90 91 

The Real Reason Behind the Polaris Success 

The authors’ supposition as to the real reason for the success of the Polaris program was the creation of 
an entirely new elite unit to manage the program, called the Special Projects Office (SPO). To motivate 
the project team they wore special uniforms, worked 5 Vi days a week, had all of their mail shipped 
“High Priority,” and were told to “think bis or get out” and unusually, enjoyed first-class travel 
arrangements and hassle free expense accounts. The SPO had a special slush fund to pursue outside 
suggestions and pacify scientific and/or academic doubters. Because schedules were critical, the SPO 
demanded and received personal pledges from contractors, and contractor's employees that their portion 
of the work would be completed on time. 

The programs headquarters occupied very modest offices in Washington DC with the goal of being seen 
as frugal by Congress. In hopes of gamering long-term and wide political support, the SBO often 
forfeited short term gains to avoid making enemies of the program. In this vein, the program often 
avoided taking the lead role in every project. They focused instead on integrating all of the components. 
The SBO always developed two or three options for every critical component, which allowed them to 
choose the best solution between the competing options. Further, they always employed a fall back 
strategy in case all three options failed. 

The SBO was very receptive to innovative ideas and actively sought out honest progress reporting of 
goals both up and down the entire organization. What commenced in 1955 with 90 people ultimately 
grew to 1,800 team members. Tremendous pressure was applied with the goal of total honesty in 
progress reporting. Lying was a cardinal crime that carried severe consequences. Tight centralized 
control was avoided to minimize deception and personnel talent was not wasted in bureaucratic 
paperwork. 

88 The Polaris System Development, Harvey Sapolsky, 1972 

89 The Polaris System Development, Harvey Sapolsky, 1972 

90 Project Management Demystified: Today's tools and techniques, by Geoff Reiss 1995 

91 A Management Guide To Pert/CPM, Jerome West 8c F. Levy 1969 
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Ramifications for failure were swift and sure. SBO branch heads were dismissed within 24 hours for 
failure, and contractor personnel were prohibited from returning to their previous positions if they failed. 
The program required long hours, and strained the staff. Promotions and awards were liberally granted 
for meeting or beating project deadlines. 


Five Possible Methodologies 

Figure 38 shows a notional comparison of four methods to determine estimate confidence levels. These 
approaches could be used to account for the unknown events in any development discipline. 



Figure 39: - Notional Comparison of Approaches 


Status Quo (Expert Judgment Method) 

Risk Identification & Tracking Method 

Approach one on the above chart is the Risk Identification & Tracking Method. Often relying on expert 
judgment, it attempts to quantify all potential risks and enter them into a software risk tracking package 
such as CxIRMA. 

Pros of the Risk Identification Software Tracking Method: 

• It is quasi defensible. 

• It will increase required reserves. 

Cons of the Risk Identification Software Tracking Method: 

• This approach places a tremendous burden on already strained resources since each risk 
specified must be “touched” regularly. 

• It is open to arguments, speculation, and manipulation. 

Estimates Occurring Early in the Process 

Early estimates of important parameters are inaccurate in two respects. First, such estimates are strongly 
biased toward over optimism. Secondly, compounding the "yes we can" unrealistic problem, there are 
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secondary cognitive errors on the part of the estimators that demonstrate departure from sound, 
evaluative thinking. 

Often it is not what we know that will get us. It isn't even what we know that we don’t know that bodes 
trouble. It is what we don’t know that we don’t know that hoses things up. This is a cognitive blindspot 
created by the fundamental nature of knowledge that has not yet been encountered. 

We focus on what we know. We look at facts based on our previous experience. We create mental 
models based on those things we know. The curse is that the smarter we become and the more we 
know, the more likely we will become self confident, and then arrogant about what we know. When we 
become excessively confident about our knowledge, the blind spot increases in size. When we become 
arrogant we stop listening and the bigger our blind spot becomes. In reality, there is not much difference 
between over confidence and arrogance. 

History is littered with examples of brilliant, yet arrogant, people who through they knew what was and 
was not, possible. One of the most accomplished scientists of the 19th Century was Lord Kelvin, 
President of the Royal Society, the United Kingdom’s national academy of science. Lord Kelvin 
defined the absolute temperature scale (named after him), created the first physics laboratory at a British 
university, conducted research leading to the second law of thermodynamics, championed the undersea 
cable, introduced Bell’s telephone to Britain, published more than 600 scientific papers, and filed 70 
patents. In 1895, Lord Kelvin declared: “Heavier than air flying machines are impossible.” He was 
subsequently proven wrong one year later in 1896 when Samuel Langley flew an unmanned steam 
powered aircraft nearly a mile. 92 

Admittedly, some probable "events" are difficult to quantify; but difficult or not, the need to quantify 
them still remains. A best effort attempt must be performed with the object of arriving at a quantified 
number as close to its future actuality as possible. 

During the early project stages (especially for technologically advanced projects) cost estimators may 
have limited relevant data available. In this case, cost estimators often turn to subject-matter experts to 
help subjectively estimate costs and variability of key drivers. This process requires an expert to specify 
key parts of the distribution for one or more inputs. Experts may be asked to supply numbers according 
to their view of what the maximum, minimum, and most likely values will be with respect to the 
activity/process/event in question. However, regardless of how expert a human being is, judgment in a 
given area is still subject to one's own biases; these "expert subjective determinations" can impose 
inaccuracies on the probability distributions derived from the expert’s inputs. 93 Generally experts place 
too narrow of a range on a possible problem. Further, biases can be intensified by the way the polling is 
done, to include 1) how the questions are phrased, 2) the order in which the questions are presented, and 
3) the amount and type of pre-judgment feedback given to the expert about the implications of the 
judgments being requested. 

A significant 94 body of psychological research done at Harvard suggests that people have a poor 
appreciation for numerical differences in magnitude of risks. Further, people's willingness to pay for 
risk reduction is usually much less than the proportional risk reduction benefit to be gained. The fact is 
that people are notoriously poor at understanding probabilities. 

Anchoring Bias 

There have been many studies on the effectfs) of anchoring bias. One 95 such study had subjects spin a 
wheel of fortune. Prior to spinning the wheel, the subjects first looked at the number on the wheel which 

92 The Vision for Space Exploration and the Retirement of the Baby Boomers, The Space Review 

93 RAND Impossible certainty : cost risk analysis for Air Force systems, Mark V. Arena 2006 

94 Willingness to Pay for Health Protection: Inadequate Sensitivity to Probability?, James Hammitt, John Graham, 1999 

95 The Black Swan: The Impact of the Highly Improbable Nassim Nicholas Taleb 2007 
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was randomly chosen. They were then asked to estimate the number of African countries in the United 
Nations. Those who were shown a low number on the wheel estimated a low number, and inversely 
those with a high number on the wheel estimated a high number. Caution in divulging specifics is 
advised when questioning experts to avoid inadvertently skewing their answers. 

Experts Have a High Opinion of Their Own Opinions 

Experts often have an unjustified bias in favor of their own opinions. They are much tougher when 
critiquing the validity of information that undercuts one of their currently held theories than they are in 
supporting information that apparently endorses one of their own tenets. Thus employing this double 
standard, they tend to dismiss new information that doesn’t fit with what they already believe. The 
danger which this self-styled arrogance creates can be illustrated by a 1982 experiment by Harvard 
researchers Albert and Raiffa. 

"Our brains have evolved (enough) to get us out of the rain, find where the berries are and 
keep us from getting killed. Our brains did not evolve (sufficiently) to help us grasp really 
large numbers or to look at things in a hundred thousand dimensions." 

Ronald L. Graham - Mathematician 


Researchers took a room full of Harvard Business School students and asked each to independently 
estimate a high/low range of numbers that answered a series of questions in such a way that they 
believed they had a 98 percent chance of being correct, and a less than 2 percent chance of being wrong. 
In other words, whatever they were guessing about had a 2 percent chance of falling outside their 
arbitrarily selected range. For example, “I am 98 percent confident that the population of Vermont is 
between 2 million and 20 million. ” The students were free to set their range as wide as they wanted. 
The experiment was not designed to test their subject matter knowledge, but rather to test their 
confidence levels of their own knowledge (e.g. to what extent they feel certain that they know 
something). The students failed miserably with an error rate on the series of questions close to 45%. 

This experiment has subsequently been replicated dozens of times, across various populations, 
professions, and cultures. The results regarding self-supporting gravitational thinking are always the 
same. The expected error rate of 2 percent is actually on average between 15-30 percent! The 
conclusion is that we over -estimate what we really know and under -estimate the possibility of our being 
wrong. Our tendency to think well of our own notions can have dramatic results in the negative. 

Philip Tetlock’s 1999 study 96 of twenty seven thousand predictions by nearly 300 experts who believed 
their predictions were narrowly bounded is highly instructive. The results of the study did not vindicate 
their "expertness" as was anticipated. The error rates were many times what these experts had predicted. 
Intriguingly, Tetlock found that there was no differential advantage for those experts holding graduate 
degrees versus those with only undergraduate degrees. His primary intent was not to demonstrate the 
competence level of experts, but rather to investigate why the experts themselves did not recognize that 
they were not as proficient in their judgments as they had supposed themselves to be. 

The point here is NOT that we should become chronically suspect of "expert" opinion nor discount its 
validity when acquiring subject matter expert inputs for project estimation purposes. Rather, the 
conclusion is that in addition to our reliance on experts, other estimating factors must be present in our 
equation in order to build an accurate joint cost/schedule calculator. 


96 Expert Political Judgment How Good is it? How Can we Know? Philip Tetlock 2005 
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Friedman (Economist) Method 

Milton Friedman, a Nobel winning economist penned an essay in 1966 titled “The Methodology of 
Positive Economics”. This essay presents a theory often cited by modem economists that loosely states 
- realism of assumptions is irrelevant as long as the theory predicts well, since the only goal of economic 
theories is to predict. This theory has been tested by Spyros Makridakis who set up competitions 
between forecasters who practiced econometrics. He compiled experts’ forecasts of real life events and 
then he evaluated the accuracy of those forecasts. He reached the conclusion that “statistically 
sophisticated or complex methods don't provide more accurate forecasts than simpler ones (methods).” 

"Everything should be made as simple as possible, but not simpler." 

Albert Einstein 

The Friedman (Economist) Method represents a second approach to acknowledging that an inductive 
event will (or at least could) happen. 

Pros of the Friedman (Economist) Method: 

• It is fast. 

• It is better than no method at all. 

• It can be implemented to affect confidence levels only above the 65% level, so reserve levels 
are not affected. 

Cons of the Friedman (Economist) Method: 

• It is Indefensible - other than the Friedman theory. 

• It is open to arguments, speculation, and manipulation. 

This Friedman (Economist) method could be utilized using a combination normal and discrete 
distribution. The discrete probabilities are set for a 30% likelihood that an unplanned event will occur. 
This will render confidence level distribution with fat tails. In the world of probability, this means that 
cost estimate "potentials" will be higher and schedule estimate "potentials" will be longer. The net effect 
is that the probability of a cost/schedule estimate overrunning the 95% mark will decrease. The 
following chart shows two "S" curves. The blue line is a "normal" distribution. The red line is a discrete 
(Friedman type) distribution. Note the minimum, maximum, mean, and standard deviation data for both 
models shown at the right of the chart. It will be noted that the Friedman distribution reflects 
substantially higher cost numbers. 
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Figure 40: - Friedman Style Approach 


Expand the High Cost Method 

Approach three is the Expand the High Cost Method. It ignores the elephant in the room and notionally 
increases the worst case event for all modeled risks. It is subjectively arbitrary and claims nothing more. 

Pros of the Notional Method: 

• It is fast. 

• It is better than no method at all. 

• It will increase required reserves. 

Cons of the Notional Method: 

• It is indefensible. 

• It is open to arguments, speculation, and manipulation. 

Quantify Everything Method 

Approach four is the Quantify Everything Method. It attempts to list and quantifies every conceivable 
risk, no matter how unlikely, (internal and external) and place a best available risk distribution on each. 

"Anything you need to quantify can be measured in some way that is superior to not 
measuring it at all." 

Gilt's Law 

NASA’s safety and mission assurance office (SM&A) spends a large quantity of resources to actively 
track and manage risks with likelihood of occurrences of one in several thousand. Not only are these 
risks tracked, millions of dollars are spent on engineering solutions to mitigate these risks. 

A recent example is an emergency egress system (EES). EESs have been installed at every U.S. launch 
pad built for manned space flight - but have never been used. NASA is currently designing a new roller 
coaster EES at launch complex 39 B with estimated life cycle costs of $40 to $80 million dollars for 
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design, build and operation. This system has a calculated probably of use of 1 time in 2,000 launches, 
and a probability that it will be successful in carrying an injured person away from the pad of 1 in 4,000. 
Despite these low probabilities, a strong case was made to add another $15 to $30 million of life cycle 
costs to double the length of the EES to allow immediate access to medical care. This additional feature 
of the EES was not approved due to lack of funding. This example stands in stark contrast to cost and 
schedule risk management mitigation events that for some reason must show a much higher probability 
of occurring before they are accepted and reflected as a risk worthy of tracking. 

Pros of the Quantify Everything Method: 

• It is quasi defensible. 

• It is a much better approach than what is currently used. 

• It will increase required reserves. 

Cons of the Quantify Everything Method: 

• This approach is very labor and time intensive. 

• It is open to arguments, speculation, and manipulation. 

JCL-PC Method 

Approach five is the JCL-PC method. Introducing this method is the primary objective of this paper. 
JCL-PC is the recommended solution set in answer to the sophisticated challenges confronting the cost 
estimating community within NASA. In essence, the JCL-PC method achieves a quantify Everything 
Method, but does so in a mathematical way, bypassing the time/labor intensive effort connected with 
method four. It anticipates 99% of the conceivable risks (internal and external), no matter how unlikely 
they are to occur and provides the decision maker with a best available risk distribution scenario. 

A full explanation of the JCL-PC method will be presented on page 85 in the "Path Forward" Section of 
this paper. 

Pros of the JCL-PC Method: 

• It is historically defensible. 

• It is a much better approach than any method currently used. 

• It is not labor and time intensive. 

• It will increase required reserves. 

• It anticipates 99% of conceivable risks and gives the decision maker a best available scenario. 

Cons of the JCL-PC Method: 

• A 1% likelihood of unknown event possibilities still remains 

A 1981 RAND Study 

A 1 98 1 RAND Study titled Understating Cost Growth and Performance Shortfalls in Pioneer Process 
Plants, reviewed 106 estimates for 40 large projects and examined the reasons for cost growth. 
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Figure 41: - Reasons for Understating Cost Growth and Performance Shortfalls 


“These initial estimates were on average very poor predictors of actual costs. Moreover, 
the range of mis-estimation was so large that it would have been difficult to apply a simple 
factoring ratio to correct for the underestimation. ” 

RAND excludes "scope growth" on the following basis; 

“One of the most commonly cited causes of cost growth is a gap between planned and 
actual scope of the plant. A clear definition of the plant's scope provides the first critical 
infonvation necessary for accurate cost estimation, and changes in scope can be a 
leading cause of misestimating. Because 'scope changes' can be variously defined, 
however, people often seize on the term and use it retrospectively as an amorphous 
catch-all to explain away cost growth. We use the term here to encompass only what the 
plant will produce and at what rate. We define scope changes, in other words, to include 
only discretionary changes in the plant's design capacity or product slate. 

During the course of a project, plant scope may be changed for a variety of reasons, such 
as changes in expected market conditions; if it is changed by altering the plant's design 
capacity or product slate, cost estimates must be adjusted accordingly. Defined in this 
manner, scope changes are largely exogenous, or external, to the accuracy of an 

estimate. 

On the other hand, cost growth frequently occurs as more precise design information is 
obtained during a project, particularly for pioneer processes. Strictly speaking, we do 
NOT define these as scope changes. They are rarely discretionary, but result from 
previously unrecognized design requirements. As engineering design progresses, more 
detailed process requirements become plain and often require additional investment in 
equipment specifications or process configurations not anticipated earlier. " 

Example Risks 

What follows over the next number of pages is a short list of possible risks, some admittedly unlikely, 
with easily available historical data that does allow bounding of the event. However, we believe most 
fall well within the 1 in 2,000 probability of occurrence. 
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Labor Strikes 

Influence from labor unions has waned in recent decades but they may again regain strength before 
program completion. Historically, strike durations at KSC have ranged between 1 and 154 days. Strikes 
may involve one or more unions and work slowdowns surrounding the strike often have negative impact 
both before and after the actual strike event. Sometimes non striking unions show solidarity by refusing 
to cross a picket line which, of course, magnifies the event. Available data was reviewed to compile the 
following list of past strikes. Many more strikes were averted through last minute agreements. 

Apollo Period Strikes 

"Work stoppages resulted in a total loss of 87,374 man-days at Cape Canaveral during a 4 1/2-year 
period in the late 1950s and early 1960s. Wildcat strikes, slowdowns, and a deliberate policy of low 
productivity further delayed progress." (Source: Moonport) 

• November 14, 1960 -KSC 

• October 31,1 964 - Hold Down Post Supplier Plant 

• February 1964 - Railroad Workers 

• April 1964 - VAB Steel Supplier 

• June 3, 1964 -KSC 

• June 8, 1964 -KSC 

• September 16, 1965 - Boeing Nationwide 

Shuttle Period Strikes 

• 1971 -KSC 

• October 4, 1977 -KSC 

• November 10, 1977 - KSC 

• 1978 -KSC 

• February 27, 1980 -KSC 

• November 1 980 - KSC 

• February 1981 - KSC 

• July 6, 1984 -KSC 

• March 1, 1985 -KSC 

• July 6, 1984 -KSC 

• November 2005 - KSC 

• June 2007 -KSC 

Weather 

NASA schedules and cost estimates are based on normal weather patterns and assume no hurricanes or 
tornados directly impacting a NASA center, or for that matter anywhere in their region. 

In the event of a direct hit to a center. Congress will likely allocate emergency repair funding - but this 
may prove insufficient to mitigate impacts to new or ongoing projects. Demand surge, is a term that 
describes the cost and schedule impacts imparted by a natural disaster. Due to the sheer volume of work 
to be done repairing the aftermath of the storm, labor and material shortages often abound. Result, a 
weather impact anywhere in the region of a center may affect projects in work or the entire programs 
cost and schedules through secondary impact. 
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Probability of a KSC Hurricane Strike 

Hurricane season occurs from June 1 to November 30, peaking sharply in early September. 

A great deal of research has been done on the probability of a hurricane landfall in the KSC vicinity. 
This data can be examined at: http://www.e-transit.org/hurricane/welcome.html . Summary analysis 
provides the following probabilities of a hurricane strike in Brevard County: 

• Tropical storm force (40-75 mph winds) 

a. 1 .8% direct hit in 2008 

b. 5 1 .5% of direct hit 50 year period 

c. 14.9% vicinity hit in 2008 

• Hurricane force hit (>75 mph winds) 

a. 0.4% of direct hit in 2008 

b. 16.3% of direct hit 50 year period 

c. 3.9% of vicinity hit in 2008 

• Intense hurricane force hit (>1 15 mph winds) 

a. 0. 1 % of direct hit in 2008 

b. 2.9% of direct hit 50 year period 

c. 0.7% of vicinity hit in 2008 

Hurricane Impacts Without Making Landfall 

In the event of a hurricane or tropical force wind event anywhere near a NASA center with the 
likelihood of landfall within 72 hours, requires termination of all work along with the implementation of 
detailed protection plan activities. Man-hour losses would be significant. 

Hurricane Impacts Making Landfall Anywhere in the Southeastern US 

Demand surge occurs whenever building materials and construction contractors are in short supply 
following one or more major catastrophes. The 2004 hurricane season saw a high demand surge when 
Florida and other Southeastern states experienced four hurricanes within 37 days. These storms resulted 
in over 2 million claims, nearly three times the volume resulting from Hurricane Andrew in 1992. This 
led to a surge in demand for materials and labor. Restrictions placed on out-of-state contractors further 
limited the supply of labor in Florida and exacerbated the recovery effort. 

In “super catastrophes” like Hurricane Katrina, losses become nonlinear, i.e., the scale of the event itself 
causes losses to increase still further. Demand surge encompasses all those elements of the costs that are 
resource-constrained. Demand surge, itself a function of the overall economic loss, introduces a 
nonlinear feedback in the estimation of economic loss. Nonlinear behaviors of economic shocks are 
described by Leontief input-output economic models. 

Building Damage 

The cost of damage incurred by a hurricane is an inverse square to the multiplier of wind speed and 
exponentially related to the number of hours with winds measured more than 50 knots. This is a 
consequence of a cyclic loading of materials which results in building fatigue. 97 


97 Dr. Bob Bailey of ASB Consulting 
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We have developed a damage model that estimated facility repair cost at both Stennis & Michoud Space 
Center with ~8% accuracy using only basic data hours after Katrina's landfall. All of the inputs events 
could be modeled and used to determine ultimate risk exposure. 


Schedule Slips 

The unexpected has a one sided effect when considering projects. It pushes in a single direction 
outward, resulting in higher costs and a longer time to completion. 

The default for cost and schedule variances is not necessarily additive. Schedule variances, however, 
can become cost variances as additional work (overtime) is often required to regain schedule, or if the 
delay creates a schedule slip. 

On an established project, the only thing more costly than stretching the schedule is accelerating it. 
Compressing the timeline is itself the most costly action known to man. 98 

A review of Kennedy Space Center 197 historical Construction of Facilities (CoF) and 1,250 Ground 
Support Equipment (GSE or STE) projects indicate a wide range of data dispersion. This subject 
appears to be complex enough to warrant additional study. However, a quick analysis reveals that the 
average and median completion dates are always beyond originally scheduled durations, sometimes well 
beyond. These values were calculated with the formula; 

Project Close out Date - NTP = Actual Days 

Actual Days Original Contract Duration - 1 = % Change from Original 


The boxplot below shows data distribution of schedule slips for various project types & durations. 
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Figure 42: - Boxplot comparing schedule slips of various types of GSE & Facility projects 


"It always takes longer than you expect, even if you take Hofstadter*s Law into account." 

Hofstadter's Law 


98 Augustine's Laws, 6th Edition Norman R. Augustine 
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Schedule slips affect cost. With cost plus contractors, the correlation between cost and schedule is fairly 
high at ~70%. On fixed price contracts, schedule slips affect cost to a much lower extent, nevertheless, 
oversight cost for FTE and WYE personnel must still be accounted for. In addition, if the project 
procurement date is delayed, additional escalation for the "slip period" must also be accounted for. 

Validation, verification & activation activities often uncover supplemental operational requirements that 
need to be resolved or capability brought on line that was previously unknown. This is the time and 
these are the people that have to make what was delivered "really" work - the tail end of the dog. 
Validation - verification activity is a prime category for schedule slip - and for that matter cost growth. 

Sundry Work Interferences with Cost and Schedule Impact 

There are a number of affiliated impacts which can affect project schedule and costs. Some of the more 
prominent potential interferences are discussed below. 

Bid Protests 

It is not uncommon for work schedules to be delayed due to bid protests following contract award either 
for alleged mistakes in the bid or for claims of non-compliance in following competitive protocols. Bid 
protests can delay project start for 85+ days if the express resolution option is exercised and/or 120+ 
days if the normal process is used, and assuming no lawsuits result. 

Funding Timeliness 

Confidence planning assumes that funding will be received when it is required; unfortunately, this is 
often not the case. Continuing resolutions, budget cuts, delays of up to six months for receipt of funding 
at the centers, complying with CoF regulations requiring 3-8 month timeframes for re-programming/re- 
directing of monies if funding is available from other government sources - all have the potential for 
delivering serious consequences to cost and schedule. 

It has been observed that Congress will often appropriate the same amount of funding for the upcoming 
year that was approved for the prior year, plus or minus three-fourths of whatever change the 
administration is then requesting, minus 4-percent tax" It can be readily seen that this legislative 
practice can wreak fiscal havoc on dependable costing strategies with direct impact on related schedules. 

Accidents 

Accidents, which sometimes are fatal, are regretfully a side effect of large construction projects. One 
such example is Launch Complex 37 which experienced two fatalities that stopped work during the 
investigation period and negatively impacted morale. In all, 29 OSHA recordable accidents occurred 
during- 1,800, 000 work hours on the LC 37 project. 

Worker Sabotage 

Often on large projects, a few bad seeds exist in the personnel pool. These individuals can substantially 
affect progress by sabotaging projects, slowing down the work, or reporting a bomb threat. This radical 
behavior usually manifests itself when the job is nearing completion and workers are facing layoff. 

• On a past launch complex construction project there were two bomb threats as project 
completion drew nye, each with a direct cost of ~$ 125,000 (excluding cost of overhead and 
equipment) for each event in FY08 $. 

99 Augustine's Laws, 6th Edition Norman R. Augustine 
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• Sabotage costs, though difficult to quantify, can be substantial with history confirming that they 
do exist. 

Operations Impacts 

During the Apollo and Shuttle buildup periods, Kennedy Space Center was (for the most part) NOT 
operational. Since both of those development projects had priority status, scheduling milestones were 
more easily achieved. This same development emphasis will not be the case during Constellation 
readying activity. The shuttle program will continue to require full center support until 2010 and 
possibly longer and Shuttle operations historically have priority when scheduling conflicts occur. This 
affects outage windows and will, to some extent, prevent efficient work progress on Constellation. 
Preparation work on Launch Complex 39B has already been delayed as a result of the Hubble Space 
Telescope's final maintenance/upgrade mission. 

The USA VAB door refurbishment project was substantially delayed because of operations impacts 
with project costs increasing from 9 million dollars to over 56 million dollars. 

Launch days can also affect delivery of equipment and/or materials for other programs making access to 
certain sensitive areas difficult and sometimes impossible. In the event of a "scrub" for the mission, 4 to 
20 days per year of lost productivity for hundreds or thousands of people can result. Since launch days 
are often a moving target, a further unproductive element of uncertainty exists for those trying to plan 
deliveries and events. The reality is that schedules often slip so fixing firm dates becomes problematic 
and further scheduling delays frequently occur. 

Late Delivery of GFE or Facilities 

Specialized components are often ordered as Government Furnished Equipment (GFE). This is done for 
two primary reasons. The first is to save money with economies of scale ordering and the second is to 
shorten project schedules due to the government placing orders that can precede the letting of a given 
contract. Long lead times can thereby be mitigated. However, once we opt for GFE involvement, if we 
fail to provide parts or equipment to the contractor in a timely fashion in accordance with contract 
requirements, the contractor has a just claim against cost and schedule increases. 

Additional slow-downs can also occur when the government has contracted with multiple firms on the 
same end product. In such cases, the first contractor must complete their part of the work before the 
second contractor can start. If the first contractor is delayed for any reason, the second contractor must 
wait until the preliminary work has been completed, and is eligible for a claim. 

Helium Shortage 

Helium is a vital commodity in the space program with unique properties and many uses including 
purging hydrogen systems, spin starting the RS68 engines, and cooling cryogenic tanks. Helium can not 
be replaced by other gasses in most instances. Because of its unique properties Helium is very 
expensive to store, handle and transport. Virtually all helium is processed and shipped to its final user as 
soon as it is extracted from the ground. To compound the problem, scientific research has multiplied the 
uses of helium over the past 50 years. It is needed to make computer microchips, flat-panel displays, 
and fiber optics. Helium is required to operate magnetic resonance imaging (MRI, scans) and used in 
many welding processes and in super-conductor research. The technology explosion is sucking up 
helium supplies at dizzying rates. U.S. helium demand is up more than 80% in the past two decades and 
is growing at more than 20% annually in developing regions such as Asia. 

Helium is found in varying concentrations in the world's natural-gas deposits and is separated in a 
special refining process. The easiest-to-get helium supplies have been tapped and are declining. 
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Supplies in the world's largest helium reserve near Amarillo, Texas, are expected to run out in six years. 
Stored in a depleted natural-gas cavern known as the Bush Dome near Amarillo; it supplies 35% of the 
helium consumed in the world. The U.S. government started the reserve in 1925, but by the mid-90s 
decided to sell it to pay off debt it incurred from stockpiling helium over the years. Under law, the entire 
contents of the Bush Dome should be sold by 20 15. 10 Once the Bush Dome reserve is gone, there will 
be no stored helium to supply the market in case of disruptions at production facilities, making for spotty 
deliveries and even higher prices. The Constellation program will require two to three times KSC’s 
current helium storage and pumping capacity. The expected helium shortage and resultant price hikes 
will likely ding the budgets by requiring a larger helium storage system, bigger commodity budgets, and 
possibly an expensive helium recovery system. 



Figure 43: - Helium annual cost change 


Space Junk 

We often think of the “vacuum of space” as a vast nothingness. In reality, space is more like a typical 
teenager's bedroom - full of clutter from the past. It's been estimated there are millions of pieces of man 
made debris circling the earth. Tracking networks only monitor about 18,000 of the largest objects 
measuring four inches or larger. This debris, which remains in orbit for many years, carries the force of 
a hurtling 400 pound safe. 

"There is a whole class of particles that can’t be tracked, can’t be shielded against, and are 
very dangerous." 

David Wright - Director Global Security Program 

In February 2009, an accidental collision occurred between an Iridium 33 Communications Satellite and 
a Russian Cosmos 225 1 . This is the ninth significant crack up in two years. Each prior event left behind 
thousands of fragments in an environment where any debris larger than a pea can be devastating. Even a 
paint flake can crack the shuttle’s windshield. 101 

"A crash (with another satellite) wasn’t on the top 10 list of problems . . . not even the top 
150 list." 

Liz Decastro - Iridium Spokesperson 

In February 2008, a U.S. Navy ship launched a missile that took out a dying, hazardous spy satellite. 
The test boosted the credibility of missile defense advocates. In 2007, China destroyed one of its own 


100 Rising Demand Makes for Helium Shortage, WSJ, 12/5/07 
,01 Harmless Debris on Earth is Devastating in Orbit, WSJ, 2-27-09 
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defunct satellites with a ballistic missile. Russia recently announced they are working to develop anti- 
satellite weapons to match efforts by other nations, 102 The Russian effort which will culminate in a 
weapons test, leaving even more debris. 

"Many spacecraft including the space shuttle are in some places venerable to particles five 
millimeters in size or below." 

Nicholas Johnson — NASA Orbital Debris Office 

In the 1980’s it was estimated that the chance of a satellite collision with space junk or with other 
satellites was one in bazillions (1 followed by 30 or 35 zeros). Since that time, the collisions that have 
occurred along with additional launches have lopped off about 20 of those zeros 103 . This trend will 
continue and will ultimately grow to become a major risk in the years ahead. 

Counterfeit Parts 

According to a recent US study, trade in counterfeit items has increased from $5.5 Billion in 1982 to 
$600 billion in 2008. Counterfeit equipment recently discovered includes aerospace components 
embossed with logos from prominent aerospace manufacturers. In 2008, NASA found a counterfeit part 
on the Kepler spacecraft that resulted in a nine-month delay in launching the unmanned probe. Kepler 
(which was finally launched in March of 2009) is designed to probe for earth-like planets in our region 
of the Milky Way Galaxy. 104 Failure caused by a counterfeit part could ultimately result in loss of 
mission or loss of crew. 

"Some of the cost overruns besetting the space agency stem from counterfeit parts 
inadvertently installed on space craft." 

Christopher Scolese - Acting NASA Administrator 


Unfunded Mandates 

New unfunded mandates are fairly common in government projects. These mandates are usually sold as 
being low or no cost. The reality is they often grow cost and push schedules to the right. Some 
examples are: 

• Conversion from US to metric 

a. Appears to increase facility costs by ~5% 

b. Higher likelihood of errors being missed during design reviews. A prime example of a 
catastrophic metric oversight was the Hubble Telescope when it was initially activated in 
orbit and began sending back blurred images due a mistake in fabricating the mirror 
curvature which resulted from failure to convert measurements from US to metric. 

• Leadership in Energy and Environmental Design (LEED) Requirements 

a. Cost impact can vary substantially depending on project size & type 

b. The plan is to continually increase certification standards 

• Environment (Other) 


102 Russia Building Anti-Satellite Weapons, Associated Press, 3-5-09 

103 The Odds, When Birds, Subs, and Satellites Collide, WSJ, 3/4/09 

104 NASA Official Says Counterfeit Parts a Growing Problem, Houston Chronicle, 3-5-09 
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a. Historical preservation 

b. Endangered species acts 

c. Wetlands mitigation 

d. Storm-water retention 

e. Green fuels for rockets & spacecraft 

i. Hypergols, perchlorate, greenhouse gasses & carbon taxes 

f. Impacts due to global climate change issues 

i. Chemicals outlawed 

ii. Commodities impacts - hydrogen, oxygen, methane, etc. 

Escalation / Inflation 

Inflationary movement during the first quarter of 2009 isn’t as yet apparent, but recent stimulus 
packages and the multiple bailouts agenda totaling $3-$8+ trillion dollars by the time it's all done will 
mean a serious influx of money in the system. Monetary escalation at this aggregate level and rapid 
pace has never been seen before and promises to create a classic inflation scenario. 

Numerous issues are at play; exchange rates, energy costs, business cycles, government regulations, 
hedge fund influences, etc. The net effect is proving difficult for even the financial experts to predict. 
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Maximum amount Current amount 


Federal Reserve - 

$5,255 trillion - 

62% 

Commercial Paper Funding Facility LLC (CPFF) 

1,800,000,000,000 

270,879,000,006 

Term Auction Facility (TAF) 

900,000,000,000 

415,302,000,000 

Other Assets 

601,963,000,000 

601,963,000,000 

Money Market Investor Funding Facility (MMIFF) 

540,000,000,000 

0 

Unnamed MBS Program announced 11/25/08 

500,000,000,000 

0 

Term Securities Lending Facility (TSLF) 

250,000,000,000 

190,200,000,000 

Term Asset Backed Securities Loan Facility (TALF) 

200,000,000,000 

0 

Other Credit Extensions (AIG) 

122,800,000,000 

122,800,000,000 

Unnamed GSE Program announced 11/25/08 

100,000,000,000 


Primary Credit Discount 

92,600,000,000 

92,600,000,000 

ABCP Money Market Fund Liquidity Facility (AMLF) 

61,900,000,000 

61,900,000,000 

Primary Dealer and Others (PDCF) 

46,611,000,000 

46,611,000,000 

Net Portfolio Maiden Lane LLC (Bear Sterns) 

28,800,000,000 

26,900,000,000 

Securities Lending Overnight 

10,300,000,000 

10,300,000,000 

Secondary Credit 

118,000,000 

118,000,000 

Federal Deposit Insurance Corporation - 

$1,788 trillion * 

■21% 

FDIC Liquidity Guarantees 

1,400,000,000,000 

0 

Loan Guarantee to Citigroup* 

249,300,000,000 

249,300,000,000 

Loan Guarantee to Lending arm of General Electric 

139,000,000,000 

139,000,000,000 

Treasury Department - 

$1.15 trillion - 

13.5% 

Troubled Asset Relief Program (TARP) 

700,000,000,000 

350,000,000,000 

Fannie Mae / Freddie Mac Bailout 

200,000,000,000 

0 

Stimulus Package 

168,000,000,000 

168,000,000,000 

Treasury Exchange Stabilization Fund (ESF) 

50,000,000,000 

50,000,000,000 

Tax breaks for banks 

29,000,000,000 

29,000,000,000 

Federal Housing Administration - 

$300 billion - 

3.5% 

Hope For Homeowners 

300,000,000,000 

300,000,000,000 

Total - 100% 

8,490,392,000,000 

3,124,873,000,000 


Figure 44: - Bailout Range as of Dec 2 2008. 


As of 3/31/09 The U.S. government and the Federal Reserve have spent, lent or committed $12.8 
trillion, an amount that approaches the value of everything produced in the country last year, to stem the 
longest recession since the 1930s. 105 

The economy is cyclical, with well defined periods of expansion and contraction, which consequently 
result in periods of high and low inflation. This cyclical pattern has been documented as far back as the 
late 1700’s. 106 

A number of studies dating back to 1752 have found that a nominal quantity of money and the inflation 
level are closely related. Milton Friedman stated in 1963 “Inflation is always and everywhere a 
monetary phenomenon." There is little disagreement that the long term correlation between monetary 
supply and inflation is very strong. However, correlation is lower over the short term. 107 


“The current efforts to help revive the economy are likely to produce inflation that could be 
worse than what the country suffered in the late 1970s.” 

Warren Buffett (2009) 


105 Financial Rescue Nears GDP as Pledges Top $12.8 Trillion (Updatel), Bloomberg.com, 3/31/09 

106 Measuring Business Cycles, Bums and Mitchell, 1946 

107 Federal Reserve Bank Are Money Growth and Inflation Still Related? Second Quarter 1999 
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Figure 45: - Correlation between the Money Supply and Annual Inflation Rates from 1953-1997 108 

Escalation rates used in NASA estimates are set at 5% for CoF and 3% for R&D projects. These rates 
are not currently modeled, and could vary substantially. The 2009 version of the NASA New Start 
index projects future annual inflation rates from 2.6%-2.9%, despite the fact that an analysis of the actual 
data contained in the index from 1959-2009 (50 years) tell us that these projections only provide a 
probability from 1 0%- 1 5% that they will not be overrun in a given year. In other words, there is an 85- 
90% probability we are not using nearly enough escalation in our calculations. To obtain a 50% 
confidence level a projected annual inflation rate of 4.6% per year is needed even without the severe 
fiscal manipulation activities currently in work by the Federal Reserve and the Treasury Department. 
Factor in the recent government intervention wild card and things could heat up to a much higher 
inflation rate with 6-7% per year not being an unrealistic projection. 

"Without a change of policy (in government's pay-as-you-go health and retirement 
system), a higher rate of inflation can be anticipated in the United States." 

Alan Greenspan (2007) 




Figure 46:- NASA New Start Index 1959-2009 probability of an overrun for a given percentage 


106 Derived from data from Federal Reserve Bank Money Growth and Inflation in the United States April 2001 
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ENR - CCI Annual Escalation Rate (%) 
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1968-1983 

6.1 

6.4 

6.6 

6.8 

7.0 

7.1 

7.3 

7.4 

7.6 

7.7 

7.9 

8.1 

8.3 

8.5 

8.7 

9.0 

9.3 

9.8 

10.6 

-) 1984-2003 

1.3 

1.6 

1.7 

1.9 

2.0 

2.1 

2.2 

2.2 

2.3 

2.4 

2.5 

2.6 

2.7 

2.8 

2.9 

3.0 

3.2 

3.4 

3.8 

-6— 1940-2006 

1.6 

2.0 

2.3 

2.6 

2.9 

3.2 

3.5 

3.8 

4.1 

4.4 

4.7 

5.1 

5.5 

5.9 

6.5 

7.1 

7.9 

8.9 

10.7 

-♦-1915-2006 

4.8 

-2.4 

-1.0 

0.1 

1.0 

1.8 

2.5 

3.2 

3.8 

4.4 

5.0 

5.7 

6.3 

7.1 

7.8 

8.7 

9.8 

11.3 

13.6 

-*—1904-2006 Normal 

-7.9 

-5.3 

-3.6 

-2.1 

■0.9 

0.1 

1.1 

2.1 

3.0 

3.9 

4.8 

5.8 

6.7 

7.7 

8.8 

10.0 

11.4 

13.2 

15.8 

1904-2006 Logistic 

-5.7 

-3.2 

-1.7 

•0.6 

0.4 

1.2 

2.0 

2.7 

3.4 

4.0 

4.7 

5.4 

6.1 

6.9 

7.7 

8.6 

9.8 

11.3 

13.8 


Probability of NOT Exceeding 

Figure 47: - Construction Cost Escalation Probabilities 


Figure 48 contains the @ Risk Best Fit Distributions for the ENR Construction Cost Index and Building 
Cost Index. These equations were used to produce the graphical lines in the table above. 


Year 

CCI 

BCI 

1915-2006 

=RiskLogistic(0.04421 , 0.031153) 

=RiskLogistic(0.0371, 0.030702) 

1940-2006 

=RiskGamma(2.0559, 0.020279, RiskShift(0.0086651)) 

=Risklnvgauss(0.055277, 0.153698, RiskShift(-0.010448)) 

1968-1983 

=RiskExtv0lue(O.O73327, 0.010914) 

=RiskLogistic(0.075144, 0.016628) 

1984-2003 

=RiskLoglogistic(-0. 028251, 0.052313, 12.387) 

=RiskLognorm(0. 034878, 0.013847, RiskShift(-0.012949)) 

1904-2006 Normal 

=RiskNormal(0.039235, 0.07215) 

No Data 

1904-2006 Logistic 

=RiskLogistic(0. 040401, 0.033168) 

No Data 


Figure 48: - @ Risk Best Fit Distributions 


Scope Changes 

NASA's confidence level analysis methodology assumes that current baseline scope is the correct scope 
up through the end of construction and into the sustainment period. This potentially invalid assumption 
could be a very costly oversight. Scope changes during the design process often result in critical path 
schedule delays and retarded procurement activities. In addition, change orders nearly always equate to 
cost growth. 

Potential Items for Scope Expansion 

Most projects take an optimistic view of the all of the elements required for project success, and omit 
items that are not 100% certain to be required, but are highly likely to be needed. For example, the 
Constellation program is not currently carrying any costs for the Orbiter Processing Facility. The 
assumption has been that it will not be used. Chances are favorable that this thinking may be reversed. 
This is but one of a number of other items that could become requirements in the future. Following is a 
partial list of potential requirements. 

• OPF: The Orbiter Processing Facility may be needed for upper stage 

checkout and storage of Ares I, or Ares V. 

• ENGINE SHOP: At KSC for engine issues 
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• CRAWLERWAY : May not be able to support the weight of the Constellation Aries V 

Launch Vehicle (CALV) 


• TRANSPORTER: May not be able to support the weight of the CALV 

• CALV: The current plan is to convert the Shuttle Mobile Launcher Platform 

(MLP) to CALV usage. Weight constraints may require new platform. 


• NEW CLEAN ROOM: For CALV vertical load ofhypergols & faring installation 


• JJ RAILROAD BRIDGE: If the JJ RR Bridge were to fail prior to scheduled replacement, the 

entire program will stop. 


Requirement Changes 

We have already added many "new requirements" to our baseline. However, the following items had 
been anticipated and were included with initial estimates - but were withdrawn when we were informed 
those estimates were "too high." Some examples follow: 

• VAB : Fabricate and install new platforms in lieu of refurbishing existing 

platforms. 

• VAB: Perform life safety and fire suppression modifications. 

• MLP & PAD: Plumb for cold helium. 


• LCC: 


Add more data review rooms in the Launch Control Center. 


Change Orders Cost More 

Change orders cost more than competitive bid work. Due to schedule constraints. Constellation Project 
changes (some of which are substantial) are being planned as change orders. Historical data derived 
from evaluating 2,752 change orders determined that (on average) the negotiated cost of change orders 
was 29.7% higher than government estimates with a 5.2% margin of error. Inversely, competitively bid 
projects (over 1 million dollars in value) are (on average) 2.2% lower than government estimates 
rendering the net change order penalty (29.7% + 2.2%) = 3 1 .9 %. 

• On average, a change order submitted by a contractor is 76. 1 % higher than the 
government estimate. 

• On average, the negotiated price above the government estimate was 29.73%. 

• On average, a competitively bid (over a million dollars) project is 2.2% below the 
government estimate. 

Of course, averages only tell part of the story. The box plot below shows the probable change order 
penalty for changes over a million dollars. (The chart includes the 2.2% delta.) 
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Figure 49: - Summary of Total KSC Historical Change Order Penalty 


Design Errors 

Design errors are a common occurrence and must be considered the rule rather than the exception. We 
can continue to expect impacts to be substantial. 

Recent Example: 

The United Space Alliance (USA) design team has been working on a VAB door 
refurbishment project that required an access platform. The weight was calculated using 
STAAD software to be 145 tons. The platform was then built and hooked to the VAB 175 ton 
crane. However, the crane would not lift the platform. Weighing equipment was then brought 
in and the measured weight turned out to be 205 tons. This was a 41 .1% delta. The project 
stopped while cause was determined. The net result was a 2 month delay and a $500K to 
$1 M cost increase. The ultimate potential impacts could have been even higher. 

Security Reverberations 

Recent security regulation changes require any one holding a KSC access badge for a period of over 90 
days for any reason to undergo a full background investigation. Many construction workers and 
delivery drivers have marginal records. Complicating the problem is the fact that many are foreign bom 
and/or are illegal immigrants. The following statistics are revealing: 

• -25% of all construction workers are illegal 

Source WSJ 12-19-2006 

• 55% of all concrete workers are foreign born Hispanics 

Source WSJ 9-18-2006 

• 22% of all painters are illegal 

Source WSJ or ENR 2006 

• A 10% increase in worker supply = a 3% decline in local wages. A 50% increase in worker 
supply = a 15% decline in local wages. 

Source Harvard Immigration Economist George Boras 2006 
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Despite the prevailing policy that illegal immigrants are not to be allowed on the Center, a recent KSC 
construction project was raided twice by INS. Time and effort to excluding nonqualified workers could 
extend schedules and increase costs. 

Catastrophic Events 

Terrorist Attack Reverberations 

A major terrorist attack anywhere in the US has the potential to severely affect cost and schedule 
performance on all work performed at KSC. 

The September 1 1, 2001 terrorist attack resulted in a substantial cost and schedule impact to the LC 37 
development project. The entire project was shut down for several days due to security concerns. Then 
it was open to essential personnel only. We were required to develop and submit a plan to resume work 
that had to be approved by the base commander. Following that submission and approval process, it 
took several more days to put the plan into action. 

At that time, all workers met outside the base and were bused onto the Center and taken to their worksite 
in the morning - then bused off in the evening. The workers were paid for their travel time. Many 
additional security guards were hired to escort delivery vehicles in and out. The total direct cost in 
2001-2002 $ was ~3 million, excluding the widespread negative productivity impacts. Because workers 
were paid for 8 hours but actually worked fewer than that (~7 hours) a ~13% quantifiable loss in 
productivity was experienced. Some actual losses were higher due to unique and subtle nuances to the 
general plan. 

After this, the Anthrax biohazard events occurred later in 2001. This had less of an effect on KSC 
projects, but it did have repercussions to other NASA sites where there were several Anthrax scares. 

Ricin was the next biohazard challenge with terrorist events starting during 2003. This contaminant was 
sent to the White House, a South Carolina postal facility, and a US Senate mail room in 2004. Resulting 
filter down effects were felt here at the Center in terms of time delays which always attend elevated 
threat levels. 

Major Accidents 

"Human space flight is inherently dangerous." 

Mike Griffin, NASA Administrator 


The Space Shuttle has experienced two major incidents to date. The Apollo Program had one. Every 
major incident results in massive cost and schedule impacts. Even with the "no risk is acceptable" 
protocol endorsed by NASA in today's world, there is at least some chance of a major incident that will 
result in loss of vehicle, loss of mission, or loss of life. 

CHALLENGER Incident 

The STS 51 Space Shuttle Challenger disaster occurred on January 28, 1986 when Challenger broke 
apart 73 seconds into its flight. All seven crew members lost their lives. The Shuttle disintegrated over 
the Atlantic Ocean, off the coast of Florida, at 1 1 :39 a.m. EST. Disintegration began 73 seconds into its 
flight when an O-ring seal in its right solid rocket booster (SRB) failed at liftoff. The O-ring failure 
caused a breach in the SRB joint it was meant to seal, allowing a flare of pressurized hot gas from within 
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the solid rocket motor to impinge upon the adjacent SRB attachment hardware and external fuel tank. 
Roughly 73 seconds into launch, the SRB joint breach flare led to the severing of the right-hand SRB's 
aft attachment. This caused the structural failure of the external tank, dumping the liquid hydrogen fuel 
load all at once and causing a massive explosion as this fuel was immediately ignited by various flame 
sources surrounding the hardware. Maverick aerodynamic forces promptly broke up the orbiter 
following the loss of attitude control. 

The Challenger disaster resulted in a 32-month hiatus in the Shuttle Program. The Rogers Commission 
(appointed by President Ronald Reagan to investigate) found that NASA's organizational culture and 
decision-making processes had been a key contributing factor to the accident. NASA managers had 
known that contractor Morton Thiokol's design of the SRBs contained a potentially catastrophic flaw in 
the O-rings since 1977, but they failed to address it properly. They also disregarded warnings from 
engineers about the dangers of launching on such a cold day and had failed to adequately report these 
technical concerns to their superiors. I09 . 

The budgetary effects of the accident 1 10 can be segregated into two categories: 

• CATEGORY 1 : "Reconstitution" costs, including the expense of investigation, any shuttle 

system modifications suggested by investigation, and replacement of 
equipment lost in the accident. 

• CATEGORY 2: Those costs and savings that result from the lower flight rate forced on NASA 

by the loss of the Challenger. 

Following are NASA's preliminary estimates: Replacing the capability lost in the explosion (exclusive 
of the orbiter) through modifying the system will require $240.5 million in 1986, $245.5 million in 
1987, and related spending of a combined $205 million for future years. Of this amount, $350 million 
will be spent to modify the shuttle-$200 million in 1986 and $150 million in 1987. 

According to the Congressional Budget Office (CBO’s) preliminary analysis, the NASA cost estimates 
above may be low. Net savings in NASA operations may offset these new costs by $98 million in 1986 
and $130 million in 1987. Of this amount, $45 million in 1986 and $124 million in 1987 would be 
saved by not operating the shuttle. Additional net savings-$53 million in 1986 and $6 million in 1987- 
are estimated for research and development and for space tracking and data communications activities. 


Fiscal Years Balance to 
1986/1987 Completion 


Anomaly (Problem) Resolution 


and Corrective Action 

605 

193 

798 

Actions in Response to the 




Rogers Commission 

170 

410 

580 

Total 

775 

603 

1,378 


Figure 50 - Challenger Accident Incurred Costs (In millions of dollars of budget authority)"' 


A 1992 White House advisory committee concluded that recovering from the Challenger disaster cost 
the country $12 billion, which included the cost of building the replacement Orbiter named 
Endeavour l2 . 


109 Vice President’s Space Policy Advisory Board, A Post Cold War Assessment o/U.S. Space Policy, December 1992 

1,0 CBO Budget Effects of the Challenger Accident Staff Working Paper March 1986 

,M CBO The 1988 Budget and the Future of the NASA Program Staff Working Paper March 1987 

112 Vice President’s Space Policy Advisory Board, A Post Cold War Assessment o/U.S. Space Policy , December 1992 
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The CBO Analysis excluded consideration of longer-term budgetary issues, such as the procurement of 
a replacement orbiter. The cost of replacing the orbiter and the existing orbiter spare parts that would be 
consumed by a new orbiter is estimated by NASA to be $2.4 billion over four years. $2.6 billion 113 was 
appropriated for a new orbiter in 86-87. 

Out-year operational costs of shuttle are likely to increase each year as the accident promotes more 
conservative and costly operational procedures. An expendable launch vehicle program to fill the 
deficiency left by the loss of an orbiter could be at least as expensive-for example, the Air Force 
recently purchased 10 rockets for $2 billion, roughly the cost of one shuttle. The new orbiter. 
Endeavour, was produced for $1.8 billion" 4 11 5 dollars, 14% under its initial 2.1 billion estimate, and 
made its first flight in May 1992. 


COLUMBIA Incident 

The Space Shuttle Columbia: At approximately 9:00 a.m. EST on February 1, 2003, STS 107 broke 
apart during it's re-entry into the atmosphere while traveling at more than 12,500 miles per hour at an 
altitude of 207,000 feet. Immediately following the accident, NASA activated a contingency plan to 
preserve all information related to this flight by establishing a Mishap Investigation Team to coordinate 
the identification, retrieval, and storage of debris and any human remains. NASA also established the 
Columbia Accident Investigation Board (CAIB). NASA developed budget estimates for implementing 
the CAIB recommendations required to return the space shuttle to flight - but not for all of the CAIB 
recommendations. The NASA Budget Office provided documentary support for portions of the return 
to flight estimate but the OMB found it to be insufficient. The agency’s projected cost for returning the 
shuttle to flight was slightly over $2.2 billion. This estimate remained uncertain until the completion of 
the first shuttle missions to the International Space Station in fiscal year 2005. 116 All Space shuttle 
Missions were halted until July 26, 2005 in order for the accident investigation to proceed and for the 
resulting recommendations to be implemented. 




Fiscal Year 



Activty 

2003 

2004 

2005 

2006-2009 

Total 

Orbiter reinforced carbon-carbon inspections 

$ 2 

$ 38 

$ 7 



On-orbit thermal protection system inspection 
and extravehicular activity tile repair 

20 

68 

130 



Orbiter workforce 

- 

5 

37 



Orbiter thermal protection system hardening 

- 

28 

34 



Orbiter certification/verification 

- 

47 

26 



Orbiter other 

- 

15 

16 



External tank items (camera, bipod ramp, 

11 

114 

94 



Solid rocket booster items (bold catcher, 
camera, etc.) 

1 

8 

26 



Ground camera ascent imagery upgrade 

8 

40 

58 



Kennedy Space Center ground operations 
workforce 


32 

36 



Other (systems integration, hardware 
processing and operations systems 
verification, and space shuttle main engine 


67 

177 



St afford -Covey team 

- 

3 

1 



Total 

$ 42 

$ 465 

$ 643 

$ 1 .067 

$2,217 


Figure 51: - November 2004 Return To Flight Cost Estimate - Real year dollars in millions 1 1 7 


1.3 CBO The 1988 Budget and the Future of the NASA Program Staff Working Paper March 1987 

1.4 GAO NASA Program Costs 1992 

1.5 Another source lists Endeavour cost as $1,897 billion, $950 million for payload bay platform, and $947 million for orbiter 

1.6 GAO Space Shuttle Costs for Hubble Servicing Mission and Implementation of Safety Recommendations November 2004 
117 GAO Space Shuttle Costs for Hubble Servicing Mission and Implementation of Safety Recommendations November 2004 
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Fiscal Year 





Percent 

Review Status 

2003 

2004 

2005 

2006 

2007 

2008 

2009 

Total 

of total 

Control board review 
complete; directive issued 

$ 31 

$319 

$117 

$ 53 

$ 47 

$ 49 

$ 39 

$ 616 

29 

Been to control board; 
directive not yet issued 

11 

146 

217 

117 

125 

134 

109 

750 

39 

In review process 

- 


309 

162 

84 

79 

70 

634 

32 

Total Return to Flight 
activities 

$ 42 

$465 

$643 

$331 

$257 

$261 

$218 

$2,217 

100 


Figure 52: - November 2004 Return To Flight Cost Estimate - Real year dollars in millions 118 
NASA’s current aggregate total for Columbia's Retum-To-Flight now stands at over $2.2 billion 1 19 

APOLLO Incident 

On January 27, 1967, the three-man crew of the first Apollo mission died when a fire erupted in their 
Apollo command module during a pre-launch test. The three astronauts were Virgil “Gus” Grissom, 
Edward White, and Roger Chaffee. A NASA investigation determined that electrical arcing in the 
modules wiring caused the fire in the oxygen rich environment. Modifications were made to the Apollo 
design and test procedures before Apollo flights resumed 2 1 months later. Cost and schedule impacts in 
total are not known. 

USSR (Multiple Incidents) 

At least four Soviet cosmonauts have perished during spaceflights. 

Cosmonaut Vladimir Komarov died during the first Soyuz flight on April 24, 1969. The spacecraft’s 
parachute tangled during descent and Soyuz 1 struck the ground with great force, killing Colonel 
Komarov. Soviet human spaceflights were suspended for 18 months while the Soviets investigated and 
remedied the problem. 

Three cosmonauts died aboard Soyuz 1 1 on June 29, 1971 when an improperly sealed valve allowed the 
spacecraft’s atmosphere to vent into space. The cosmonauts - Georgiy Dobrovolskiy, Vladislav Volkov, 
and Viktor Patsayev - were not wearing spacesuits and were asphyxiated. There were no Soviet human 
spaceflights for 27 months while modifications were made to the spacecraft. 

There have been serious problems during re-entry of Soyuz capsules. In April 2008, the Soyuz capsule 
missed its landing area for the third time since 2003 due to equipment malfunctions. Investigators 
suspect that the ballistic re-entry was caused by an electrical short in the cable that connects the crew 
capsule's control panel with the Soyuz descent hardware. 

Information on the replacement cost and loss of operations costs for these combined USSR incidents is 
not available. 

Large Scale Catastrophic Events 

The above items are all somewhat "likely" to occur. Events such as the foregoing relate directly to 
mission fulfillment activities and mission execution comes with little to no margin for error in pursuit of 
success. Failures, oversights, and emergencies in technical affairs won't generally terminate program 
continuance. It is safe to predict that the work will go forward notwithstanding costly incidents and 
unwanted consequences from time to time. 

1,8 GAO Space Shuttle Costs for Hubble Servicing Mission and Implementation of Safety Recommendations November 2004 
119 NASA’s Implementation Plan for Space Shuttle Return to Flight and Beyond, June 2006 
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There are, however other "non-space" events that are far less likely to occur but which would certainly 
have more far reaching results. These types of events would be of such a magnitude that program(s) 
could be shelved completely and/or be so severely affected that recovery will never take place. These 
kinds of major catastrophe events must not be excluded from analysis. They include such things as: 

• World War Three 

• A pandemic (worldwide disease event) 

• A nuclear detonation on American soil 

• The collapse of the US economy or a major restructuring of government's infrastructure 

• A catastrophic event on the Shuttle Program, the Constellation Crew Launch Vehicle 
(CLV) or Cargo Launch Vehicle (CALV) during pre-launch processing, launch, mission, 
or landing 

• An administrative decision preventing or delaying continuance for any reason(s) 

Examples from History 

The Ripple Effect - A Case Study 

Often, changes to a prevailing design or existing contract have an unintended ripple effect. Launch 
Complex 37 was designed using the best available information - but it was not good enough! Due to 
new requirements, the Mobile Service Tower (MST) needed additions to various systems. The 
aggregate effect of these additions/changes was that the steel support structure became overloaded. The 
structure was summarily reinforced (at great expense) to support the extra loads. The total weight 
increase to the structure was 24.7%. It was then determined that the concrete bridge over the flame 
trench (installed 2 years previously) was not strong enough to support the now 9.6 million pound MST. 
The flame trench bridge repairs became a huge effort that diverted valuable project resources and cost 
(~$6 million dollars in 2001 $) and pushed the schedule out ~10 months (excluding the additional 
evaluation and decision time preceding the time to work the repairs). Effectively demonstrated here in 
the case of LC 37 was the cost-schedule ripple effect with its long tail of consequences. 



Figure S3:- Ripple effect repair work being accomplished at Launch Complex 37 


In summary, all of the required changes to the MST were estimated and approved. Premium prices were 
paid for each change and the schedule deviations were absorbed. Then the MST overweight impacts 
occurred and the cost and schedule consequences were again absorbed. Then, numerous contractors 
who were involved in the changes summarily submitted additional claims, which were negotiated down 
and subsequently paid. These claims included extended overheads, stacking of trades, etc. So the 
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summom bonum cost for a projected one million dollar change to the MST ultimately cost six times that 
amount when all related costs and impacts were calculated. 


POINT: If the "final" total cost for the CR described in the preceding example would have 
been known and budgeted for by those of us in Cost Estimating then submitted prior to 
CR approval, we would have been accused of padding the numbers. If our figures would 
somehow have been accepted, the cost benefit analysis may have prevented the CRs 
from being approved in the first place. The point being made here is that changes often 
cost MUCH more than initially envisioned by those responsible for completing the work. 
Often, at the beginning of a modification effort, a grossly inadequate cost ceiling is levied 
by some well meaning manager up the line. But well meaning or not, such a declaration 
doesn't make the final cost less real. It is what it is. CONCLUSION - NASA can do better 
at calculating and accepting more robust and realistic estimates and have the courage to 
hold to those estimates in the public forum. Not only CAN we do this - it is our fiscal 
responsibility to do sol 

The Orbiter Processing Facility 

The Space Shuttle's Orbiter Processing Facility is another prime example of looking at the world 
through the rose colored glasses of early estimating. Original Shuttle concepts envisioned that 
maintenance between launch events would be like those associated with the servicing of a jet bomber - 
requiring a minimum number of qualified technicians in a wide open hanger, scooting their ladders up to 
the big bird, performing the needed servicing on a weekly tum-around schedule, and then towing it out 
the hanger door and on to the launch pad ready for the next mission. 

Due to the extreme complexity of the shuttle and the fundamental differences between space flight and 
atmosphere-bound flight, there ended up to be almost no similarities between the servicing needs 
originally conceived and the actual maintenance tasks that surround a Shuttle's return to orbit. Any 
similarities between that early concept and what really evolved have long since disappeared. The 
current Orbiter Processing Facility is a honeycomb of steel platforms that surround the Shuttle and house 
sophisticated testing and refurbishing equipment in ready proximity to the world's most complicated 
space cargo vehicle. The pictures below tell the story more effectively than words. The equation for 
this story would look something like this: 

What was estimated fin 19741 © what was needed and built [today] 


Shuttle Operations Concept in 1974 The Orbiter Processing Facility Today 

This is what was estimated. This is what was needed & built. 

Figure 54: - The support scaffolding and service area that was actually required to maintain the Space Shuttle provides an 
excellent example of the inaccuracies inherent in "early" estimating. 
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The point here is that it will probably never be possible in a high tech development engineering 
environment for a team of experts to "foresee" the end result at the beginning of the development 
journey nor predict the myriad of in-work alterations that will be needed to bring a program through to 
completion. The following famous quote will have particular meaning to the NASA community: 

"I think as I write ... as I think as I write ... as I think as I write." 

Pearl Buck 

NASA people too must always think as they work in a series of repeating cycles while they move toward 
successful fulfillment of early but often inadequately defined concepts. The end result may be but a 
distant cousin to the original notion - just as the end cost and schedule may bear little resemblance to the 
earlier optimism. The challenging duty of the cost estimator is to correctly project the delta before it is 
known - a difficult task to be sure - but one that can be accomplished more reliably through the use of 
better tools. 

More NASA Troubles 

Despite many NASA successes, such as the Pathfinder and Exploration Rovers on Mars, the loss of life, 
unsuccessful missions, and unforeseen cost overruns have recently increased congressional concerns 
over the benefits of such exploration. NASA has had difficulty bringing a number of projects to 
completion, including several efforts to build a second generation of reusable human spaceflight vehicle 
to replace the space shuttle. NASA has attempted several expensive endeavors such as the National 
Aero-Space Plane, the X-33 and X-34, and the Space Launch Initiative, among others. None have 
completed their objective of fielding a new reusable space vehicle. The GAO estimated 120 in 2006 that 
these unsuccessful development efforts have cost approximately $4.8 billion since the 1980s. 

NASA has tried unsuccessfully to develop a number of vehicles to replace the shuttle over the past three 
decades. In the 1980s NASA initiated the National Aero-Space Plane (NASP) to build and test a 
manned experimental flight vehicle for demonstrating single-stage-to orbit space launch and sustained 
hypersonic cruise capability. NASA canceled the program as it was experiencing cost overruns, 
schedule delays, and technology problems. GAO reported that from 1986 to 1993 NASA spent $398 
million for the NASP program. 

In the 1990s, NASA began the X-33 program to develop single-stage-to orbit technology and the X-34 
to demonstrate reusable two-stage-to - orbit technologies. According to a 2006 Congressional Research 
Service report, NASA terminated the X-33 and X-34 in March 2001— after spending over $1.4 
billion — because the cost to complete them was too high relative to the benefits. In 1999, GAO reported 
that technical problems and unrealistic cost estimates on the X-33 project alone led to cost overruns of 
$75 million and over a year’s delay. 

In 2004, after the announcement of the Vision for Space Exploration, NASA canceled the Space Launch 
Initiative (SLI) program, which was to provide both launch capabilities and an emergency crew return 
from the ISS. NASA’s Inspector General reported that NASA did not verify and validate basic 
requirements for its second generation space transportation, while GAO reported that key management 
controls could not be implemented until such requirements were defined. GAO estimates that from 2001 
to 2005 NASA provided the SLI program with about $3 billion in funding. 


120 GAO-06-8 17R Exploration Cost and Schedule, 2006 
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Mike Griffin on ESAS Cost Analysis 

• "... cost models capture only past practices and management culture. But 
we are not sufficiently trusted - indeed, I do not trust us! - to get away with 
claiming cost “savings” on the basis of “new ways of doing business” that 
we have not yet demonstrated. It should be our purpose to change the cost 
models of the future. 

• It was my very conscious thought as we went through ESAS that, unlike in 
the early Shuttle planning, where we "costed” a Shuttle flight on the basis of 
having as many as 60 flights/year to amortize the fixed costs, we should go 
in the opposite direction with Exploration. ...no one can ever accuse us of 
upfront lowballing to "buy in" to the program. 

Figure 55: - Mike Griffin on ESAS Cost Analysis 


Constellation Program - Ares I and Ares V 

This brings us to the work at hand. Any meaningful discussion of the Constellation Program's joint 
probable cost and schedule must candidly acknowledge our track record through the historical 
perspective as discussed on the previous pages. Conclusions, determinations, and insights on how to go 
forward with improved approaches that will produce more reliable cost and schedule projections is the 
object and purpose of this paper. 

The content we have selected is meant to illustrate and substantiate a long pattern of underestimating 
both cost and schedule, not only by NASA, but by DOD and other industries. Early confidence level 
analysis that does not even consider the possibility of triple digit cost and schedule growth is willfully 
ignoring historical precedent and the full cannon of possible risk factors that show up as game-changing 
players on the field of cost estimating. 



• LES AND APOLLO-TYPE PAYLOAD 102, 875 LB 
■INSTRUMENT UNIT 3, 870 LB 


• SECOND STAGE 
(MODIFIED S-IVB) 

■ INTERSTAGE 

FIRST STAGE 
(260-IN. SRMI 


253,882 L8 
4.183LB 

3, 749, 423 LB 


TOTAL VEHICLE 

WEIGHT AT IGNITION 4,114,233 LB 


THRUST AT LIFTOFF 


5,500,000 


Figure 56: - The 1967 Concept of Ares I Vehicle with 260 ” SRB 2 (Shuttle & Ares use 156" SRB ’s) 


121 Launch Facilities and Operations for Large Solid Motors Study, Final Report Vol II DAC-58079 12-28-67 
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Encapsulated Service 
Module (ESM) Panels 


Instrument Unit 
• Primary Ares I control 
avionics system 



Orion CEV 


Upper Stage 

• 302. 2K Ibm LOX/LH 2 prop 

• 18-ft diameter 

• Aluminum-Lithium (Al-Li) structures 

• Instrument unit and interstage 

• Reaction Control System (RCS) / roll 
control for first stage flight 

• Primary Ares I control avionics system 


Interstage 


Stack Integration 

• 2.044 OK Ibm 
gross liftoff mass 

• 325.0 ft in length 


First Stage 

• Derived from current 
Shuttle RSRM/B 

• Five segments/Poly butadiene 
Acrylonitrile (PBAN) propellant 

• Recoverable 

• New forward adapter 
Avionics upgrades 



Upper Stage Engine 

• Saturn J-2 derived engine (J-2X) 

• Expendable 


Figure 57: -Ares 1 Rocket DAC 2 TR6 Overall Length 320.81 * 

In the authors opinion, many of the components of the Ares 1 launch vehicle in their assembled and 
stacked configuration are either new or require major redesign and development. The "new 
development" label is fully apropos when discussing most of the hardware associated with 
Constellation. 


Constellation ought to be benefiting from the best and most accurate cost and schedule 
projections ever submitted on a NASA program road map. It is our opinion that 
unwelcome fiscal news provided early in the process to the Congressional Budget Office 
is preferable to discovering the very same information from an after-the-fact historical 
perspective. It will save us time, money, and effort in the long run if given realistic 
numbers from the start. Eventually acknowledging the same programmatic conclusions 
when Constellation is in the history books isn't value added. Cost estimating should 
always be more about credibility and reliability up front - than about patronizing 
expectations along the way. 


In September 2004, the Congressional Budget Office analysis of NASA’s New Vision for Space 
Exploration 122 stated. 

“Costs over the 2005-2009 budget period for the lunar return mission would total $6.4 billion, 
NASA projects, all of which would be used to develop the CEV. Through 2020, total funding for 
human exploration would be about $66 billion, (2005 $) in NASA's estimation, which includes 
$2.2 billion between 2018 and 2020 for follow-on missions after the first human lunar return 
landing. (The exact content of those missions is undetermined.) With the potential exception of 
those additional funds, NASA's budget projection through 2020 does not include explicit 
development and procurement of other systems that would be necessary for establishing a 
lunar outpost or for carrying out future human missions to Mars. ” 

“Initial development of the CEV, including test flights, is expected to last from 2005 through 
2014. The first CEV test-of a partially capable prototype-is planned for 2008; the first 


122 A Budgetary Analysis of NASA’s New Vision for Space Exploration, CBO, Sept 2004 
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unmanned test of a fully capable unit is scheduled for 201 1, and the first human flight, for 2014. 
NASA's plans for both the test flights and the lunar mission are based on using an EELV to 
boost the CEV into low-Earth orbit. 

NASA's projected funding covers the procurement of four operational vehicles for about $730 
million each. Including research and development, testing, and operations, costs for the CEV 
through 2020 would total roughly $24. 7 billion. ” 

This pre-phase A data is provided as a base reference, since the program has evolved from the initial 
concept. 


NASA's Projected Budget for the First Human Return to the Moon, 2005 Through 2020 

(Billions of 2005 dollars) 

2005-2009 

2010-2020 

Total 

Crew Exploration Vehicle 

6.4 

18.3 

24.7 

Lunar Lander 

0 

13.4 

13.4 

Heavy-Lift Launch Vehicle 

0 

17.9 

17.9 

Operations 

0 

L9 

L9 

Total 

6.4 

57.5 

63.8 


Figure 58: - NASA Budget for First Human Return to the Moon 2005 Through 2020' 23 


CBO’s 2004 analysis also presumed costs for the new exploration mission grew might grow by 45%, 
based on analysis of 72 prior NASA programs. If this occurs on Constellation (and there are no 
indicators to suggest that it won't) the CBO’s analysis indicates that NASA’s total funding needs 
through 2020 will require adding $32 billion 124 to NASA’s proposed 63.8 billion budget - making the 
completion mark a $95.8 billion. However, historical data makes a strong case that more complex 
programs of longer durations (which Constellation is) experience the greatest rate of growth. The full 
range of growth in budgeted costs for the 72 programs the CBO used was -25 percent to 274 percent. 

In reference to the Bush VSE Program: ". . . one of the biggest unfunded mandates that we 

have had in all of government history." 

Former Senator John Glenn 

An additional potential for cost overrun is the unresolved, uncovered asset base (both facilities and 
personnel) during the interim period between Space Shuttle standown - but prior to re-engagement of 
those assets under the Constellation Program. An example would be a "flight controller" who will 
certainly be needed again but not for a few years. Retaining qualified people and needed facilities 
during the "dead space" could become a very costly proposition. 

In the 72-program set in the CBO analysis, average cost growth fell from 140 percent in the 1970s to 
about 20 percent in 2000. Happily, NASA’s estimates appear to be improving, but the improvement is 
measured against smaller programs. The average cost of NASA’s programs has declined from about 
$3.5 billion in the 1980s to about $500 million in the 1990s and apparent accuracy improvement may 
prove illusionary. 

The Agency’s past three decades showing ever lowering levels of cost growth may not be applicable to 
Constellation's new exploration initiative. Constellation projections should use former development 
efforts of similar size and magnitude if accurate forecasting results are hoped for. 


123 CBO A Budgetary Analysis of NASA’s New Vision for Space Exploration, 9-2004 

124 $63.8 x 145% = $28.7 The CBO’s figures apparently include some expected cost growth in other NASA’s programs 
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A 2006 GAO report 135 , is cause for chagrin at NASA HQ. They emphasize the absence of cutting edge 
proficiency in the cost estimating world within the agency when contrasted with the world renowned 
scientific and technological respect enjoyed by NASA scientists and engineers. To wit: 

“Although NASA is continuing to refine its exploration architecture cost estimates, the agency 
cannot at this time provide a firm estimate of what it will take to implement the architecture. The 
absence of firm cost estimates is mainly due to the fact that the program is in the early stages 
of its life cycle. 

According to NASA cost-estimating guidance, early life cycle phase estimates are generally 
based upon parametric models, which use data from projects with similar attributes to predict 
cost because there are usually many unknowns and actual cost or performance data are not 
available. NASA preliminarily identified the resources needed to implement the architecture as 
outlined in the architecture study primarily through the use of such models. NASA conducted a 
cost risk analysis of its preliminary estimates through fiscal year 201 1. On the basis of this 
analysis and through the addition of programmatic reserves (20 percent on all development 
and 10 percent on all production costs), NASA is 65 percent confident that the actual cost of 
the program will either meet or be less than its estimate of $31.2 billion through fiscal year 
2011 . 

For the cost estimates beyond 2011, when most of the cost risk for implementing the 
architecture will be realized, NASA has not applied a confidence level distinction. 

Since NASA released its preliminary estimates, the agency has continued to make architecture 
changes. For example, following the issuance of the architecture study, NASA conducted 
several analysis cycles during which various aspects of the architecture have evolved, such as 
the diameter of the CEV, the engine used to support the upper stage of the CLV, and the size 
of the Reusable Solid Rocket Booster on the CLV. While these changes and others are 
appropriate for this phase of the program, when concepts are still being developed, they leave 
the agency in the position of being unable to firmly identify program requirements and needed 
resources, which can also be expected at this phase of the program. 

According to NASA officials, once they receive more detailed contractor inputs, the agency will 
be able to produce higher-fidelity estimates of program cost. NASA plans to commit to a firm 
cost estimate at the preliminary design review (PDR) in 2008, when the program’s 
requirements, design, and schedule will all be baselined. NASA will be challenged to 
implement the architecture recommended within its projected budget. " 


125 GAO-06-8 17R Exploration Cost and Schedule 
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Given this budgeting challenge, let's look at the Apollo Program. The 2005 NASA Exploration Systems 
Architecture Study (ESAS) - the official phase A study - states that the historical cost in current 2005 
dollars for the Apollo Program through the first lunar landing (FY61-FY69) was approximately $165B. 
The $165B figure includes all civil service salaries and overheads along with all Government “service 
pool” costs. The ESAS architecture for Constellation has an estimated total cost of $124B through the 
first lunar landing (FY06-FY18) and takes into account anticipated productivity gains and the use of 
Shuttle derived hardware and evolved available technologies. 

With all else being equal, the NASA/Air Force Cost Model (NAFCOM) would predict that most Apollo 
flight hardware developed 50 years later would cost 33 percent less to develop and 22 percent less to 
produce than in 1967. Some subsystems (such as avionics) have an opposing trend toward increasing 
the cost over time due to increased functional requirements. This ESAS assumption is based on annual 
productivity gains of ~ 2.1%. 



AJI costs are ‘full costs” (including civil service, Government support, etc.) 


Figure 60: - Constellation's Exploration Systems Architecture Study in comparison to Apollo costs 
These credit adjustments in the foregoing analysis may not prove valid because: 

• Constellation is developing two new rockets - not one. 

• Use of Shuttle developed hardware is substantially less than anticipated in the ESAS study. 

• Apollo placed two crew members on the moon for 3 days; whereas the Ares architecture places 
four crew members on the moon for 1 days (4.6 times more working days on the lunar surface 
per sortie mission). 

• Ares architecture also calls for access to the entire lunar surface; whereas Apollo was confined 
to the equatorial regions. 

• Ares architecture will be designed for anytime return from any lunar location, thus requiring 
more sophistication and enhanced delivered capability over Apollo. 

• The Ares capsule has three times the volume of the Apollo Command Module. 

• Astronauts are no longer expendable with a 0 incident standard in place. Initial Apollo & Shuttle 
flights were launched with many items (rumored to be in the hundreds or even thousands) that 
were not cleared for flight. These required a management waiver to "go for launch. " 

• NASA escalation rates used in NAFCOM may be too low given the historical data. 
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• Legacy costs may not be fully accounted for. An example would be the unfunded mandate to 
switch all systems to S.l. units. This has the potential for massive cost increases since 
thousands of systems, drawings, specifications, and operations manuals must be changed, 
proofread, and tested if applicable to achieve the upgrade/conversion. 

• Overall technical sophistication has resulted in systems growing in complexity and redundancy 
since Apollo. 

• Parts made from composite materials and lightweight alloys are generally more costly to design 
and manufacture in comparison to unalloyed metal parts. 

Conclusions 

We have presented evidence that cost and schedule growth is pervasive and biased toward 
underestimation - and has been that way since at least the late 1 800’s. If we keep doing things the same 
way, we will keep getting the same results. Until we address and factor in all the causes of cost growth 
and schedule extension, (internal and external) the real outcomes we will inevitably continue to get, will 
dwarf not only our original estimates but also our follow-on estimates as well. 

The authors have chosen to be guided by the following axiom: 'If unwelcome (even pessimistic) opinions 
are suppressed, while optimistic projections are welcomed and rewarded, an organization's ability to 
think critically and realistically will be undermined.' The material chosen for this paper gives a 
reasonably accurate picture of the many variables involved in predicting the outcome of complex 
development projects. The truth is that estimating as a discipline, has produced a track record replete 
with inaccuracy - a fact that none of us are satisfied with. We also believe there are many who feel as 
we do - that better estimating is achievable, notwithstanding the resistance that may arise from those 
who feel we must live and die by optimistic budgets - ignoring mind the facts of history in the archives 
of the past. 

The cost-schedule paradox is a formidable opponent. Clashing forces create a volatile fulcrum (cost 
affects schedule - schedule affects cost). If funding for a given year is exceeded, that portion of the work 
that was supposed to be accomplished - can’t be done. Inversely, if the schedule slips (for any reason) 
escalation and fixed costs will continue to accrue, producing a steady rise in the price tag as it marches 
onward in time. Like interest, development costs never sleep - because time never stops. 

After a review of over 500 contracts, the Office of the Under Secretary of Defense for Acquisition has 
observed that once a contract is 15 percent complete, if it is over budget, it is highly unlikely to realign 
itself with original cost/schedule projections, and the final percent overrun will be greater than the 
percent overrun to date. Despite this fundamentally important and subsequently proven 126 
127 observation, contractor and governmental personnel often claim that THEIR programs will be 
different. To our dismay, those of us who know better have not yet found the persuasion or the 
influence to recalibrate the thinking of those who wield power in the fiscal circles of influence; nor have 
we been able to infuse the courage within them to tell it like it is and then hold their ground. The quest 
for this essential modis operandi is still out there somewhere. 

To provide trustworthy joint cost-schedule confidence level assessments, the estimator must 1) research 
and obtain the most reliable input from subject matter experts in the system or systems in question, 
2) incorporate valid mathematical risk analysis, 3) infuse perceptive and candid scheduling projections, 
and 4) make probability estimates that border on the prophetic. These combined forms of expertise are 


126 RAND Predictability of Costs, Time and Success of Development 1959 

126 Christensen, Davis S. and Kirk Payne. 1992. Cost Performance Index Stability - Fact or Fiction? Journal of Parametrics 10, (April), pp.27-40 

127 Heise, Capt Scott R. 1991. A Review of Cost Performance Index Stability. MS Thesis, AF1T/GMS/LSY/91S-12. School of Systems and Logistics, Air Force 
Institute of Technology, Wright- Patterson AFB, Ohio 
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rarely found synergistically assembled. That is, those who understand a system best in the aggregate are 
not consistently the best at expressing their expertise. 

This paper has stated that a deeply embedded leadership “culture” contributes to cost growth and 
schedule extension in the form of natural optimism by senior management. This occurs not only during 
the genesis stages of program development but often throughout all program activity stages. Most 
senior managers are “can do” type people who do not respect adequately the peripheral possibilities for 
productivity hiccups or admit to their organization achieving less than optimum performance. To fail is 
a genuine four letter word to them and rather than aggressively consider ALL the potential skeletons in 
the development closet, they opt to only "deal with those things" when and if the unwelcome events 
arise. They regard themselves as optimists who must convey that message to their "people" if they are 
to successfully give their organization the chance to succeed. Few leaders walk the walk of a realist. 

The paradigm of the joint confidence estimator is this: optimism does have its place, which place is for 
those who lead to approach the project with confidence that it CAN be achieved with caution. But it 
does NOT mean that program objectives are a given, nor can they be realized without due consideration 
for ALL the potential pitfalls that can come to pass during project fulfillment. To ignore these 
"potentials" isn't optimism - it's naivete. Whenever optimism blinds managers to those things that may 
be, they become reactive from that moment forward. They lose their proactive privileges. 
Unfortunately, most charismatic standard bearers have never learned this important leadership truth. 


"Ninety percent of the time things will turn out worse than you expect. The other 10 
percent of the time you had no right to expect" i: * as much as you got. 

Augustine's Laws, 6th Edition - Norman R. Augustine 


“We i tid repurl 
but it wo* certainly in my 
b»*t inlemtt to be very 
optimiuk about what we 
c oukido.' 

Bunn<< pum! nut on I hr Mlrtbtr 
pniffUH ani *vC I A 


Boeing ux is “unprepared to 
otmiM 1 judgment in 

certain area* because it 
u*t%n'l within their own 
exf>erienc*“ 

DONATO M Rf Ml 


"/ shouldn't have ulhuxd it 
to go forward" 

RUIN* MATT 

« pn - ) ihnlM »ln fl m r B.nli| la 
URWlhruArlrtM 


"The train unxA uvs 
predetermined on Day I. 



“We huve lost double-digit 
billion r on mulellile 
programs that ueren't 
effectively managed by the 
government" 

CMAIXTOPMf R V BONO 

fepuMtrui iwNo r (nm Mmoun 


Figure 61: - Quotes from NYT Article on NRO Satellite program Future Imagery Architecture 129 


When there is a lot of money on the table, no wants to say they can’t do it. The contractor's ethic is: win 
the program at any cost and sort it out later. Correct the government ’s sins and my sins with overruns. 

Exceptional parallels can be drawn between personal portfolio management in the news of late and 
project risk management which we have been discussing. A Wall Street Journal article 130 really puts the 
subject into perspective. The article succinctly points out how the best strategies, developed by the best 
and brightest minds, often find their way into the trash can, victims of the “human condition.” 

Examples from the article are: 

• Harry Markowitz, recipient of a Nobel Prize for his mathematical treatment of relationship 
between risk vs return in portfolio management, ignored his own prized work in managing 
his personal investments, preferring to follow an “emotional" strategy. 

• John Bogle, investment guru and retired founder of Vanguard funds, holds to the belief 
that investors should re-balance their portfolios on a regular basis. But Bogle admitted that 
he hasn’t altered the allocation in his personal holdings since March 2000. 


128 Augustine's Laws, 6th Edition Norman R. Augustine 

129 FAILURE TO LAUNCH In Death of Spy Satellite Program, Lofty Plans and Unrealistic Bids By Philip Taubman NYT 1 1-2007 

130 WSJ “Investing Experts Urge ‘Do as I Say, Not as I Do”’ Jason Zweig 1-3-09 
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• Don Phillips, managing director at Momingstar, highly recommends placing tax-inefficient 
assets in retirement accounts but never has followed his own advice and thus pays 
preventable taxes year after year. 

• Burton Malkiel, Princeton economist and author of “A Random Walk Down Wall Street, " 
and an intellectual godfather who praises the virtue of index funds, confesses that 
individual stocks comprise a substantial portion of his personal investment portfolio. 

All of these examples have one thread in common, namely: emotion trumps education . This will come 
as no surprise to those of us who are regularly engaged in risk management activities. We know full 
well how significant and serious these so called “soft issues” of risk management can become. It's not a 
difficult stretch to consider how resultant problems can expand exponentially and stymie participants in 
the world of joint confidence level analysis. 

We in the NASA Cost Estimating community can recommend the most wonderful and efficacious 
algorithms, heuristics, and quantitative techniques, much as do the experts who advise those in portfolio 
risk management - but - if the advice we provide is not followed, we have accomplished nothing except 
that of informing and educating the crew of a sinking ship - and the right to say "we told you so" when 
things start to flounder - which provides no satisfaction for anyone. Our role as joint confidence level 
analysts is intended to supply intelligence that will save the ship, not stand by and watch it list for a 
while and then capsize into budget over-runs and missed milestones in spite of the warnings we tendered 
from the beginning. Our not yet accomplished objective in the cost estimating community is to find a 
way to not only be heard by top management, but to be believed and valued as actual contributors in the 
mapping process of NASA's complex development programs. We must not be content to merely take a 
benign position as placeholder-type personnel in the organizational scheme of things. This paper is an 
attempt to move us toward this worthy objective - that of making the discipline of cost estimating a 
genuinely respected agency component that will make a difference on the bottom line of the NASA 
Budget. 

We've attempted to summarize the objectives of this paper by diverting from our traditional role as 
information advisory personnel to that of taking the liberty of critiquing managerial influence on 
organizational dynamics. We have only done so with the desire to provide valuable steering influence in 
the cost estimating genre to those who manage and guide - but not in any way to imply disrespect or cast 
dispersions toward the individuals burdened with the heavy responsibility of leadership. Mr. Jefferson 
said it well: 

"History ... by apprising (people) of the past . . . will enable them to judge the future." 

Thomas Jefferson 


Summary 

Studies indicate that the accuracy of an estimate has a direct relationship to the stage of development of 
a project. Another way of saying this is that estimates improve as development progresses. It also has 
become apparent that estimates for developmental projects containing only “modest technical 
advances”, tended to be more accurate than those projects which are more ambitious - where the 
development threshold is being pushed substantially. 

Many factors contribute to cost and schedule growth, but over-optimism in initial designs, changes in 
scope over time, inherent technical difficulty of maturing technologies, and external influences are 
common themes. NASA is not the only federal agency facing cost and schedule growth issues. The 
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causes outlined above are not unique to NASA. The NASA/DOD procurement system has been 
analyzed for decades. Commissions, panels, and academic studies report causation do to these same 
issues. The fundamental challenges of good cost and schedule estimating and work performance are 
remarkably similar across federal agencies. 

Valid JCL probabilistic estimating of cost and schedules depends on the stability and soundness of the 
baseline. Inasmuch as each project's estimate is developed with many inputs, it holds that any changes 
to those underlying inputs (assumptions) will reduce the program’s cost and schedule confidence. 
Substantial differences of opinion exist within the cost-estimating community on how to develop and 
interpret these probabilistic estimates. 

“In some cases, the content or complexity of the technical baseline is underappreciated. 

In other cases, the initial estimate of technical resources such as mass or power is 
inadequate or reliance on heritage systems is overstated. The initial inadequate 
technical baseline and/or poorly defined requirements lead to an artificially low initial 
cost estimate resulting in significant cost growth beyond the project’s internal cost 
reserves. Furthermore, optimism may be introduced into the cost estimating process 
from empirical cost models that do not incorporate cancelled missions, missions 
currently in development that are experiencing difficulties, or missions whose actual 
costs have been omitted or modified based on “unusual” circumstances. Another key 
driver of a project’s final cost is schedule risk, which is often not adequately captured, 
making the initial schedule incompatible with the budget, resulting in an overall plan that 
is not executable. In summary, the optimism in the initial design starts the cycle, which 
is exacerbated by limitations in the cost estimating process.” 

Gary P. Pulliam, Vice President Aerospace Corporation 2009 

The day of reckoning is drawing near. Bipartisan legislation ( called the Weapons Systems Acquisition 
Reform Act of 2009 ) has been introduced to Congress. It gives greater leeway to kill programs with 
unchecked cost overruns and seeks to establish realistic cost estimates. 131 Currently, Pentagon officials 
are required to notify Congress when a program goes significantly over budget - but it is relatively easy 
for the Defense Department to then obtain a waiver based on a claim of necessity. The new bill is 
designed to ensure that more pentagon contracts are fixed price awards instead of the cost plus contracts 
currently so prevalent. In the new government contracting environment, arriving at estimates that are 
very close to eventual reality will become more important than ever before. 


“Dealing with these cost and schedule issues is hard, and there’s no simple fix or the 
situation would have been resolved long ago.” 

Gabrielle Giffords, chairwoman House Subcommittee on Space in Aeronautics 


A study of historical disasters 132 suggests that the proximate cause of failure is rarely a single event; 
usually it was one small problem that precipitated a cascade of errors or unlikely events, which were 
often made worse by poorly trained human operators. In a few cases, superior training and experience 
saved the day, preventing serious technological failures from turning into headline-busting news. 


Bill Targets How U.S. Buys Weapons, WSJ 2-25-09 
132 Inviting Disaster, James Chiles, 2001 
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“Individuals, processes and organizations are perfectly designed to achieve whatever 
results they are currently getting, so if you’re not happy with what you are achieving, 
its time to reconsider your assumptions and approaches to your process and product 
design methods.’’ 

Andy Grove, Chairman of Intel 

With history as our guide, we can expect budget cuts, plus ups, continuing resolutions, strikes, natural 
disasters, and other problems that are beyond NASA’s control. We know the future will be impacted by 
improbable events, we just don’t know what or when. Lack of control in these matters, however, does 
not have to equate to the absence of contingency planning. Though the unexpected always brings with it 
unwanted impact, real confidence level assessment also includes balancing factors to offset these 
unknown risks. 


“There is significant competitive pressure, both within NASA and among its contractors, 
to initiate a mission at the lowest possible cost.” 

GaryP. Pulliam, Vice President Aerospace Corporation 


“The space acquisition system is strongly biased to produce unrealistically low cost 
estimates throughout the acquisition process; these estimates lead to unrealistic 
budgets and un-executable programs.” 

Joint Task Force on Acquisition of National Security Space Programs 

Everyone agrees that space flight is difficult. The United States and many other nations have blown up a 
lot of missiles and rockets in the past in pursuit of reaching for the stars. No one argues that failure is 
part of the business. In fact, people in all walks of life have coined a euphemism for something that is 
difficult to do. They refer to it as rocket science. One of the pervasive root causes for cost growth is the 
realization that we can’t aim for the stars unless we at least begin the journey - that we can never get 
there if we don't start. And getting started is why we deliberately estimate optimistically. 

When adequate cost and schedule reserves are available, often as through a self fulfilling prophecy 
budgets expand and schedules grow. A self fulfilling prophecy is the phenomenon that occurs when a 
given expectation evokes certain behaviors that make the expectation come to pass. "In other words, 
once an expectation is set, we tend to act in ways that are consistent with that expectation. Surprisingly 
often, the result is that the expectation, as if by magic, comes true ." 133 If people accept the notion that 
budgets and schedules have reserves available, they will go forward with the assurance that their pet 
project should be initiated, or that overruns are acceptable. At some point along the road of execution 
the reserves are expended, “on nice to haves” and reserves are not available for the “must haves”. 
Strong management resolve is required not to allow this phenomenon. 

“Invariably, the Department of Defense and the Congress end up continually shifting 
funds to and from programs - undermining well-performing programs to pay for poorly 
performing ones. At the program level, weapon system programs are initiated without 
sufficient knowledge about requirements, technology, and design maturity. Instead, 
managers rely on assumptions that are consistently too optimistic, exposing programs 
to significant and unnecessary risks and ultimately cost growth and schedule delays.” 

GAO Report 09-362T 


133 Social Theory and Social Structure, Robert Merton, 1957 
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It is critical that estimates be "real and adequate" right from the beginning and that all the players on the 
fulfillment team realize that the budget is the budget. Dedication to "living within ones means" will 
apply equally to the world of work just as it does ones private finances. 

Conversely, if a project/program has more resources on hand than it needs, management must be firm in 
reversing the old anecdote that claims projects will expand to fit the allocated time and budget. The 
"use-it-all-up" philosophy should be ferreted out and eliminated from NASA organizational behavior. 
More money can always be spent and more time can always be used. Real program needs vs. exorbitant 
wants are often separated by a thin wavering grey line. As a basic practice, management should 
frequently ask themselves the question: just how important is it to trim a day off the processing time or 
acquire this additional capability? 

As a corollary, "management reserves" should never be used to cover the cost of scope growth. The 
meaning of "reserve" is "set-aside money" which is to be held back for the purpose of 
accommodating the completion of existing program/project scope and static baselines. As a core 
guiding principle, new scope should always mean new money. 


Some uncertainties are within the realm of the project’s control. Proposers can be overly 
optimistic in their efforts to provide the most attractive package in a competition. The 
cost savings assumed based on the use of “heritage technology’’ for spacecraft or 
instruments can be over estimated. New technology development can ultimately be 
much more challenging than anticipated. Sometimes inadequate time is planned for 
early engineering efforts and refinement of requirements. These are all areas within 
project accountability.” 

Christopher Scolese - Acting NASA Administrator 


With regard to risks and planning for the unknown, it is often argued that some cost and schedule risks 
should NOT be modeled since they have not recently occurred or in some instances every happened. 
Just because we have not yet died does not mean we are immortal. Rare events (some of which may be 
catastrophic) must be included in our JCL analysis to safely claim we have provided "good" confidence 
level guidance. 


“Out of the ten NASA projects in the GAO QuickLook Report that exceeded the 
Congressionally-mandated cost and schedule thresholds, approximately half did so as a 
result of external factors : some with limited solution options open to NASA.” 

Christopher Scolese - Acting NASA Administrator 


Just as the occurrence of an unlikely event can substantially affect project cost and schedule, problems 
can arise if something minor (which was not expected to be a problem) becomes one. For example 
during the development of the SRB, it was discovered that the production of a certain type of rayon fiber 
(an ablative material) anticipated for use on the SRB, was being phased out due the demise of bias ply 
tires - something not associated with space related acquisition . 134 Mitigation resulted in a project impact. 


134 Congressional Record 1977 NASA Authorization Volume 1, Part 3 
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In recent years, NASA has commissioned several studies to determine the primary 
contributing factors to cost and schedule growth. These studies, as well as others in the 
field, identified several common themes: significant optimism in initial designs, changes 
in scope associated with the evolution of the design overtime, the inherent technical 
difficulty of developing world class technologies, and the effect of external influences . . . 

Gary P. Pulliam, Vice President Aerospace Corporation 


One final example is offered. From 1940 through the 1970’s only a few US banks failed. Twenty five 
banks failed in 2008. 135 More will fail in 2009 despite the massive federal financial bailout effort. The 
Capital Purchase Program (CPP) reports that 400 banks in 47 states have received federal funds under 
this program 136 . Many more financial institutions are on the verge of insolvency or were absorbed by 
mergers prior to the fact. Large brokerage houses and insurance companies have failed or have required 
massive bailouts to sidestep their closure. Why is this so shocking? Because it simply wasn’t supposed 
to happen! The banking industry, brokerage houses, and insurance companies employ thousands 
(probably tens of thousands) of risk experts - people with masters and PHD degrees who are there to 
secure the way ahead. These companies have spent untold millions on sophisticated risk models and 
hedging strategies. But their banks failed nonetheless due to reasons no one seemed to have power to 
prevent. Why? - Because they failed to include the possibility of the occurrence of rare events. 

The object of this paper has been to introduce and defend the following point. Claiming a specific JCL 
for a project without including consideration of 1) over-optimism in initial designs, 2) changes in scope 
over time, 3) inherent technical difficulties connected to maturing technologies, and 4) external 
influences where rare events and potential risk possibilities are included in the probabilistic assessment 
process, 5) possibility of triple digit cost growth - will ultimately fall short of its aim. That aim is to 
provide credible and accurate fiscal guidance to those who authorize the resources and those who 
oversee the fulfillment activities. Anything less than this is simply NOT acceptable. 

The Path Forward 

The acid test of whether NASA will find the institutional resolve and clarity of purpose to raise a banner 
in support of our recommendations calling for candid realism on the part of all managers at all levels of 
the joint confidence level playing field - is yet to be determined. NASA’s fiscal game plan - to tell it like 
it is on public and congressional stages - should become as transparent as it has been within the private 
offices and institutional corridors of the agency. A recent editorial from NASA's Wayne Hale is relevant 
and timely, an excerpt of Mr. Hale's article is inserted here: 

"... I have been out of the Shuttle Program manager job for almost a year now and a 
trusted coworker just a week ago told me that people in his organization had been 
prevented from giving me important alternative choices for some program choices that 
occurred a couple of years ago. This was staggering. It was happening right in front of 
me and I was totally unaware that people - whom I trusted, who I hoped would trust me - 
kept their lips sealed because somebody in their middle management made it clear to 
them that speaking up would not be good. 

"Astounding. 


135 http://www.fdic.gov/bank/individuaI/failed/banklisLhtrnl 

136 Top 20 banks receiving US aid are lending: Treasury, www.breitbart.com 2-17-09 
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"About two weeks ago an activity that Mike Coats started at JSC had an all day report out 
period. The Inclusion and Innovation Council was to propose ways to improve innovation 
at NASA. Various teams reported out, including one team of young employees who has 
the task to talk about the barriers to innovation at NASA -- specifically at JSC. 

"The video attached was their result. I found it extraordinarily funny and not at all funny. 
These young people have obviously found themselves in situations RECENTLY in which 
managers at various levels applied sociological and psychological pressures to keep them 
from bringing ideas forward. 

”1 am convinced that if we asked the managers who were the models for this little morality 
play whether they stifled dissent or welcomed alternate opinions, they would respond that 
they were welcoming and encouraging. Probably because they have that self image. 

"But actual behavior, not inaccurate self perception, is what we really need. . . 

”1 feel like the early civil rights pioneers must feel; the overt bad behavior is gone 
underground. People say the right things in public discussion of how they should act, then 
behave in the bad old ways in small or private settings. 

"Since these behaviors are still being practiced at NASA, here is what I believe managers 
need to do. 

"1. Break out of the sandbox. Even if it is not your area, the agency needs the best ideas 
(in order) to succeed in our goals. If you have subordinates who have ideas for improving 
other areas, it is important to get those ideas into the open where they can compete in the 
marketplace of ideas, or at least get a technical review. 

”2. If a subordinate has an idea that has been tried before and didn't work, consider that 
times may have changed and it might work now or (might work) with improvements that 
you know of. In the final extremity, your subordinate needs more than the curt dismissal 
that it's been tried before and didn't work - you need to explain it to them. 

"3. Managers at all levels need to provide safe places and times for interaction that skips 
levels in the chain of command. 

"Well, that is enough to start with. Looks like we still have a long way to go and the first 
step is to know that you still have a problem." 

'Wayne NaCe, NASA Shuttle Manager 

As authors of this work, we find ourselves in the immensely important but unenviable crucible of 
proposing a hard doctrine to those above us. The challenge of correcting a cost and schedules estimating 
culture that predominantly has held to the hazardous precedent of over-optimizing performance 
projections is a daunting foe indeed. However, in the hope that intelligent dissent is valued NOW more 
than ever at NASA, we are prepared to raise the bar on the joint confidence level concept - believing that 
a new day in business practices at NASA will genuinely welcome a new way. Changing the landscape 
has never been easy, but when a "better way" is finally endorsed, it is always worth it! 


The (JCL-PC) Method 

We introduce here a landscape changing JCL equation that mathematically compensates for the 
optimism bias inherently present in NASA cost estimating activity. It is called the Joint Confidence 
Level - Probabilistic Calculator (JCL-PC) and will correct the overly optimistic cost and schedule 
estimates that have long plagued the NASA cost estimating community. The JCL-PC will safeguard the 
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agency against submitting dangerously optimistic numbers with the resultant loss of credibility for 
NASA. Using the JCL-PC will avert the consequences of missed cost targets where plus-ups have 
become a way of life within the agency. We propose that several key individuals at NASA HQ be 
trained in the use of the tool. These individuals would become the "window" for the submits coming 
from the various NASA Centers prior to forwarding funding requests to the Congressional Budget 
Office (CBO). We are confident this process will produce numbers that prove out much closer to reality 
when viewed from the vantage point of history. 

The JCL-PC is a holistic algorithm that effectively compensates (at any stage of a program or project) 
for the unidentified risk events that have not yet occurred or cannot yet be acknowledged or discovered. 
As a program/project progresses forward and a risk item moves from a "potential" event to being a 
"discrete" entry in the baseline, the JCL-PC equation will become less weighted in direct proportion to 
actual work performed and actual schedule achieved. In this way, the JCL-PC is a fully cooperative 
player with earned value concepts. In short, the JCL-PC is the self-adjusting constant in the cost 
estimating law of relativity. It achieves mathematically what human monitoring cannot yet determine 
and foreshadows illusive fiscal requirements and schedule adjustments so important to good cost 
estimating. The JCL-PC is the missing link to realistic budgets and PPBE planning. 

Bayesian Statistics Growing Out of Elicitation 

Elicitation of subjective probability distributions is an area of research arising from several 
developments in the 1950s and 1960s. The first was a renewed interest in Bayesian statistics. 
Practitioners of Bayesian statistics argued that probability should reflect a subjective state of knowledge 
in which a rational person would use Bayes’ theorem to modify an initial state of knowledge about some 
item of interest. In our case, that happens to be the final cost/schedule of our programs. So ... we start 
with the initial probability distribution, and then modify it with acquired data to form an updated 
probability distribution (referred to as the posterior probability distribution). This argument is 
contrasted with the prevailing school of Frequentist statistics which holds that the only basis for 
assigning probability to an event was its random chance of occurring in a large run of experiments. 
Bayesian statistics, therefore, requires two steps: 1) eliciting the prior distribution and then 2) performing 
the mathematical calculations required to apply Bayes’ theorem in pursuit of a more accurate prognosis. 

A Capsulated View of The JCL-PC 

The JCL-PC estimating method is based on the hypotheses that in the beginning phases of a project 
there are many unknown risks - and over time the project will have a high probability of exceeding 
estimated costs and scheduled duration. Examples of these risks have been discussed in detail in the 
preceding pages. As the program/project progresses, optimism biases will fade and hard realities will set 
in. It is then that time and money resources must be applied to effectively deal with those unforeseen 
events. Work as it was initially planned will inevitably change. Quantifiable risks become clearer and 
NASA’s S-Curves will tend to lay down as the work goes forward. Keep in mind that it's not the project 
that is becoming inherently riskier. It's a matter of participants fully identifying the real work that was 
"out there" all along. Even though the scope of the work wasn't fully perceived "back when" - progress 
has continued to identify the risks and quantify the corrective actions. History is written in real time and 
that history differs to a greater or lesser degree from what was anticipated. The JCL-PC helps us better 
plan for and manage that difference. 

"While useful and necessary for the initial planning phase of a mission, early estimates 

are, at best, educated guesses made with preliminary conceptual information." 

Christopher Scolese - Acting NASA Administrator 
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The JCL-PC method strikes a needed balance between subjectivity and anticipated risk variability 
leaving only one remaining probability influence factor to deal with. - namely, assigning the percentage 
complete of the subject project. This % complete factor includes both subjective and objective elements. 


"The devil is in the assumptions. Often in engineering reliability one has sufficient 
experience to make the model accurate. However in financial or aviation models it is 
extremely difficult to get the assumptions right. All the money and expertise on Wall 
Street was incapable of producing models that accurately predicted the risk that banks 
were running when buying complicated financial instruments." 

David McDonald - Mathematician 


The proposed project is generally not what is finally built. Likewise, the initial cost estimate is not 
representative of the final, as-built configuration. Required changes were unavoidable as understanding 
of the design evolved. In essence, cost estimators are trying to estimate a moving target as projects 
progress toward their final design form. 

A recent study of 40 NASA science missions (which are relativity small projects) found only 12.5% did 
not experience any cost or schedule growth above the project's internal reserves. 137 Surprisingly, this 
result is close to the construction industry mega project average where 14% of projects were completed 
for the original cost estimate. 1 38 

"Unanticipated risks often manifest themselves late in the development cycle during 
integration and test, when it is often too late to make adjustments." 

Gary P. Pulliam, Vice President Aerospace Corporation 


Several NASA research efforts with respect to Figure 62, confirms that the Agency’s cost and schedule 
performance is better when measured from the KDP-C gate than when measured from the earlier 
milestones. 139 This supports our aforementioned hypothesis and the body of historical data that 
estimates improve with project progression. Since available data suggests that cost estimates are more 
susceptible to large overruns than schedule estimates, our example will be built on cost - but the same 
methodology is applicable for schedule estimating. 


137 Gary P. Pulliam, Vice President Aerospace Corporation, Congressional Testimony Subcommittee on Space and Aeronautics, 3-5-09 

,3 * Megaprojects and Risk: An anatomy of Ambition, by Bent Flyvbjerg, Nils Bruzclius, and Werner Rothcngatter (Cambridge University Press, 2003). 

139 Christopher Scolese, Acting NASA Administrator, Congressional Testimony Subcommittee on Space and Aeronautics, 3-5-09 
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NASA's Life Cycle for Flight Systems and Ground Support Projects through Phase D 
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SDR = System Definition Review PDR = Preliminary Design Review 

NAR = Non-Advocate Review CDR = Critical Design Review 

Figure 62: - NASA defined life cycle phases through Phase D 


Figure 63 below shows the cost growth for 96 historical projects, from the first available estimate to the 
last available cost estimate. The average growth was 93%, the median growth was 51%. 
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Figure 63: - Historical percentage cost growth for 96 NASA projects 


Shockingly - these numbers are probably low! Since many of these projects are still in development. 
Projects are often re-baselined multiple times - as shown in Figure 64 below. Occasionally, projects are 
canceled due to cost growth. But here's the telling point. Growth is historically calculated against the 
latest baseline - NOT against the initial baseline. So ... if cost growth were computed against the initial 
baseline, numbers would be more aggressive and margins considerably greater. 

Although Figure 63 is the basis of our example, the 96 aggregate projects may not be applicable due to 
Simpson's Paradox. It's a well accepted rule of thumb within that the larger the data set, the more 
reliable the conclusions. Simpson's paradox, however, is a contradiction to that rule. Caution must be 
used when amalgamating data from multiple projects types because conclusions can be flawed. This is 
caused by lurking variables or skewed data from unequal sized groups being combined into a single data 
set which distorts the output Therefore, when using the JCL-PC method, care must be taken to only 
compare “like projects” when estimating probable cost and schedule growth. 
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Then-year dollars in millions 



The best way to explain this is with a simplified example. When examining world track and field 
records we see that for the 100 Meter Dash, the record is 9.77 seconds. The world mark for the 
Marathon is 2 hours, 4 minutes and 55 seconds (7,495 seconds). The average for time for both races is 
3,752 seconds. Therefore, the expected time for the next 100 Meter Dash is 3,752 seconds and the 
expected time for the next Marathon is also 3, 752 seconds. Statistically impressive, however, is the 
expected time for the next Half Marathon - which is also 3, 752 seconds and wouldn't you know it, that 
time is close to the record for a Half Marathon (record 3,500 seconds). The point is easy enough to see. 
Terribly inaccurate projections could be made from the data without conscientiously allowing ONLY 
"like races" in the evaluation set. 

Simpson's paradox should be respected when applying the JCL-PC method and making business 
decisions in the NASA environment. Results may be deeply misleading if the schedule slip of a small 
planetary mission was used to project the schedule slip for the development of a new rocket system that 
will place a manned craft near the polar regions of the moon. Schedule slips are not feasible on most 
planetary missions since they will miss a launch window that may not come again for many months or 
years. Inversely, schedule may well be traded for cost reductions on large projects where schedule can 
be reasonably modified. Firmly holding to a particular launch window in such an instance may not be 
critical to mission fulfillment. 

Notionally, "like missions" in the NASA environment could be segregated as follows: 1) Planetary, 2) 
Satellite, 3) Manned Space Flight, and 4) Other. Further data analysis should determine the proper 
allocations. 

Figure 66 demonstrates the JCL-PC estimate multipliers for various probability ranges. Basically, it 
adjusts for what we know to be true - that early estimates are optimistic and that the project is very likely 
to overrun. The red line shows that at the beginning of a project if you want to achieve a 40-50% 
confidence probability, the costs will be ~1.6 times the estimated costs. However, as the project 
progresses and unknown risks become more fully identified - resulting in optimism bias decreases - then 
at a 40% project completion point, the 50% estimate is considered to truly have a 50% probability. An 
ever decreasing amount of unknown risks, however, will still remain until the project is completed. 
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Figure 65: - JCL-PC multiplier applied at various project phases 


To demonstrate the concept Figure 67 shows an S-Curve comparison for a fictional $1 million dollar 
project point estimate, with no other risks induced. 


0.843 1.598 



_ Project @ 0% 
Complete 

Project @ 

— 25% 
Complete 

Project @ 

— 50% 
Complete 

Project @ 

— 75% 
Complete 

Project @ 

— 100 % 
Complete 


Figure 66: - Example project with JCL-PC applied to typical point estimate 


Figure 68 shows the S-Curves for the same $1 million dollar project, but includes the JCL-PC optimism 
corrector and some minor project risk, through a more typical project life cycle with project scope creep 
causing final cost to increase to $1.66 million or 1.66 times the original estimate, (59% probability point 
for historical aerospace project growth). As the project evolves the S-Curve moves slightly to the right 
and becomes more and more vertical. 
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0.71 1.71 



Figure 67: - Example project with JCL-PC applied to typical Confidence Level estimate 

The notational method of accomplishing this is as follows. 


Assumes no 
Unknown 
Risks Occur 

Unknown Risk Distributions 

% Project 
Comp 

% Project 
Remaining 

JCL-PC Multiplier 

1.00 

=RiskBetaGeneral(0.98679,26.007,- 
0.1 6733, 29.928)+1 

0% 

100% 

=RiskDiscrete(A9:B9,C9:D9) 


Table 2: - Notational method for JCL-PC applied to typical Confidence Level estimate 


JCL-PC = Project cost estimate x Discrete Risk Distribution 

• Discrete Risk Inputs: 

o 1 - Project percent complete = Project percent remaining 
o NASA's historical cost growth 
o Constant of 1 

• Probability of cost growth decreases as project progresses 


Assumes no 
Unknown 
Risks Occur 

Unknown 

Risk 

Distributions 

% Project 
Comp 

% Project 
Remaining 

JCL-PC 

Multiplier 

1.00 

1.93 

0% 

100% 

1.93 

1.00 

1.93 

100% 

0% 

1.00 


Table 3: - JCL-PC multiplier at 0% and 100% project complete stages 

In actual practice it is difficult to ascertain the exact percentage of project completion. So it is 
recommended that the project percent complete be replaced by a uniform distribution with a range of 
completion percentages. For example if a project is between 10% and 20% complete 
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Figure 68: - Uniform distribution, suggested input for project percent complete 


0.73 1.52 



Figure 69: - Comparison of Status Quo JCL method vs. JCL-PC method 


All examples of JCL-PC have been done using cost probabilities. The same method is applicable for 
schedule estimates as well, but for brevity purposes they are not presented in this paper. 

Figure 68 displays the estimated cost of a one million dollar project at 50%, 60%, 70%, and 80% 
confidence levels at project completion marks ranging from 1% to 100% as derived using the Joint 
Confidence Level Probabilistic Calculator. There are some meaningful observations that can be made as 
we examine the chart. 

Observation 1 : At 0% complete, the numbers across the row to the right reveal varying 

financial requirements needed by management in order to feel progressively 
greater levels of confidence that the task at hand can be accomplished with 
the stated amount of funding. 

Observation 2: When the project is 50% complete, you'll notice that a 50% confidence level 

suggests that the project can be completed for the anticipated $1 ,000,000. 
However, if we adhere to the NASA standard of a 70% confidence level, we 
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see that another $400,000+ will likely be needed to complete the project. No 
matter how well a project is managed, it rarely compensates for ultra- 
optimistic budget estimates that sooner or later return with a vengeance and 
overcome the most skillful leaders. 

Observation 3: The whole intent of the JCL-PC method is to scope "accurately" the lay-of- 

the-land in the program/project fulfillment world. The JCL-PC method does 
not "pad the budget" in any way. To the contrary, it strikes a fair and 
equitable balance for both the CBO (who has a need to know that supplied 
monies will actually accomplish the intended goals) and the NASA 
performing organization (who deserves sufficient resources to get the job 
done without cutting safety and quality comers.) The JCL-PC is the long 
awaited neutral facilitator who has the best interests of all parties at heart. 


As bleak as the previous cost growth analysis is, it pales when compared to development cost growth for 
manned space flight projects. These projects are typically large, complex, long duration, and strive for 
leaps in technology innovation. Historically, all factors correlated with large cost growth percentages. 


0.36X 1.601 



Figure 70: - Manned space flight development project growth compared to all 
NASA development project growth 


The JCL-PC methodology should be employed at headquarters level to prevent project advocates from 
gaming the system and submitting an unrealistically low estimate or squandering essential reserves. 
Headquarters should be the holder of reserves and projects should be held accountable for overruns and 
be required to come to Headquarters to ask for additional funding above the initial request. Reserves 
should never be regarded as a slush-fund cushion that compensates for inefficiency or poor management 
- but should be available for valid and unforeseen working developments as intended. 


“NASA needs to surgically remove many of our administrative processes & reports from 
off the shoulders of management. Through using the JCL-PC Method, optimism bias will 
become a thing of the past. Estimates will be credible, budgets will be respected and 
adhered to, and managers will genuinely be able to work their projects "as planned" 
notwithstanding unexpected events. Through using this approach, NASA will see the 
improvements it has been waiting for.” 

Glenn Butts - 2009 
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% Project 
Comp 

50% Probability 

60% Probability 

70% Probability 

80% Probability 


$ 1,609,066 

$ 1,857,133 

$ 2,174,399 

$ 2,616,440 


$ 1,598,078 

$ 1,846,323 

$ 2,163,685 

$ 2,605,307 


$ 1,585,663 

$ 1,834,122 

$ 2,152,228 

$ 2,594,747 


$ 1,574,433 

$ 1,822,371 

$ 2,140,592 

$ 2,583,494 


$ 1,563,509 

$ 1,811,140 

$ 2,129,614 

$ 2,572,611 


$ 1,552,785 

$ 1,801,310 

$ 2,119,332 

$ 2,561,769 


$ 1,541,261 

$ 1,789,630 

$ 2,108,168 

$ 2,550,626 


$ 1,529,168 

$ 1,777,058 

$ 2,095,463 

$ 2,537,132 


$ 1,516,338 

$ 1,765,030 

$ 2,082,426 

$ 2,525,748 


$ 1,503,448 

$ 1,751,744 

$ 2,069,885 

$ 2,513,173 


$ 1,489,706 

$ 1,737,258 

$ 2,055,196 

$ 2,500,522 


$ 1,475,736 

$ 1,724,750 

$ 2,042,870 

$ 2,484,315 


$ 1,465,601 

$ 1,714,396 

$ 2,032,325 

$ 2,473,910 


$ 1,452,985 

$ 1,702,363 

$ 2,021,113 

$ 2,463,222 


$ 1,440,544 

$ 1,689,323 

$ 2,009,071 

$ 2,455,022 


$ 1,426,756 

$ 1,676,526 

$ 1,995,655 

$ 2,441,035 


$ 1,412,468 

$ 1,662,142 

$ 1,980,270 

$ 2,426,189 


$ 1,399,755 

$ 1,648,626 

$ 1,965,857 

$ 2,410,808 


$ 1,386,037 

$ 1,635,652 

$ 1,954,296 

$ 2,399,669 


$ 1,372,947 

$ 1,621,448 

$ 1,939,293 

$ 2,382,456 


$ 1,360,167 

$ 1,609,409 

$ 1,928,335 

$ 2,372,409 


$ 1,344,635 

$ 1,593,364 

$ 1,911,705 

$ 2,357,853 


$ 1,333,685 

$ 1,583,432 

$ 1,902,295 

$ 2,347,507 


$ 1,318,294 

$ 1,567,763 

$ 1,891,420 

$ 2,336,302 


$ 1,303,174 

$ 1,554,213 

$ 1,871,708 

$ 2,320,607 


$ 1,289,200 

$ 1,537,384 

$ 1,858,291 

$ 2,305,082 


$ 1,272,705 

$ 1,522,836 

$ 1,845,361 

$ 2,290,987 


$ 1,254,900 

$ 1,505,320 

$ 1,826,851 

$ 2,279,456 


$ 1,241,265 

$ 1,494,088 

$ 1,813,336 

$ 2,258,111 


$ 1,225,943 

$ 1,476,599 

$ 1,796,962 

$ 2,246,298 


$ 1,208,380 

$ 1,459.973 

$ 1,782,621 

$ 2,223,345 


$ 1,194,153 

$ 1,445,453 

$ 1,765,732 

$ 2,211,968 


$ 1,180,633 

$ 1,433,033 

$ 1,753,182 

$ 2,201,219 


$ 1,159,391 

$ 1,415,927 

$ 1,736,025 

$ 2,180,268 


$ 1,145,376 

$ 1,395,525 

$ 1,718,317 

$ 2,167,650 


$ 1,125,722 

$ 1,377,224 

$ 1,697,959 

$ 2,152,046 


$ 1,109,768 

$ 1,361,297 

$ 1,681,585 

$ 2,126,428 


$ 1,089,703 

$ 1,340,298 

$ 1,663,528 

$ 2,111,215 


$ 1,073,074 

$ 1,321,185 

$ 1,644,121 

$ 2,097,397 


$ 1,055,352 

$ 1,307,108 

$ 1,633,245 

$ 2,082,129 


$ 1,036,021 

$ 1,288,094 

$ 1,612,696 

$ 2,064,563 


$ 1,016,947 

$ 1,266,124 

$ 1,585,794 

$ 2,038,385 


$ 1,000,000 

$ 1,244,597 

$ 1,566,420 

$ 2,017,618 


$ 1,000,000 

$ 1,231,943 

$ 1,554,578 

$ 2,001,107 


$ 1,000,000 

$ 1,208,197 

$ 1,531,093 

$ 1,984,033 


$ 1,000,000 

$ 1,188,041 

$ 1,510,236 

$ 1,960,073 


$ 1,000,000 

$ 1,165,292 

$ 1,485,487 

$ 1,941,414 


$ 1,000,000 

$ 1,150,335 

$ 1,472,031 

$ 1,923,773 


$ 1,000,000 

$ 1,125,387 

$ 1,450,458 

$ 1,904,659 


$ 1,000,000 

$ 1,105,308 

$ 1,427,268 

$ 1,881,346 




•/o Project 
Comp 

50% Probability 

60% Probability 

70% Probability 

80% Probability 


$ 

1,000,000 

$ 

1,057,800 

$ 

1,385,759 

$ 

1,835,546 


$ 

1,000,000 

$ 

1,036,906 

$ 

1,359,787 

$ 

1,811,089 


$ 

1,000,000 

$ 

1,012,134 

$ 

1,335,648 

$ 

1,786,727 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,306,103 

$ 

1,763,228 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,290,673 

$ 

1,740,835 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,265,746 

$ 

1,718,179 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,232,599 

$ 

1,685,812 


$ 

1,000,000 

s 

1,000,000 

$ 

1,215,361 

s 

1,668,453 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,184,725 

$ 

1,636,432 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,156,708 

$ 

1,605,007 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,123,590 

$ 

1,576,575 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,098,902 

$ 

1,554,492 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,067,209 

$ 

1.517,164 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,035,774 

$ 

1,485,793 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,003,519 

$ 

1,459,294 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 

s 

1,428,904 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 

$ 

1,387,431 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 

$ 

1,354,001 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 

$ 

1,323,170 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 

$ 

1,288,540 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 

$ 

1,248,773 


$ 

1,000,000 

s 

1,000,000 

$ 

1,000,000 

$ 

1,207,219 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 

$ 

1,170,704 


s 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 

$ 

1,129,256 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 

$ 

1,080,471 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 

$ 

1,039,946 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 


$ 

1,000,000 

$ 
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$ 
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$ 

1,000,000 

$ 
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$ 

1,000,000 

$ 

1,000,000 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 


$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 

$ 

1,000,000 
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1,000,000 

$ 

1,000,000 
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1,000,000 

$ 

1,000,000 
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1,000,000 
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1,000,000 
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1,000,000 
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1,000,000 
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1,000,000 

$ 

1,000,000 


$ 

1,000,000 
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1,000,000 

$ 

1,000,000 
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1,000,000 


$ 
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1,000,000 
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1,000,000 
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Figure 71: - Numerical cost estimate output for a 1 million dollar project at four different JCL percentages and project 
completion stages ranging from 0% to 100% using the JCL-PC method. 


Specific Recommendations 

If NASA really wants accurate joint cost and schedule confidence level results data, these are the things 
we must do. 

# 

> IN A POLICY STATEMENT: 

1) Clearly specify that all risks (internal and external) are to be included in JCL 
analysis. If a hundred risks each have a 1% probability of occurring - in theory 
there is a 1 00% chance that at least one of the risks will occur. If the low-likelihood 
risk impact is large enough, it can materially affect project outcome. One method is 
to use the JCL-PC estimating equation introduced in this paper. 
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2 ) Mandate precise criteria upon which we are being asked to provide JCL 
assessments, (i.e. X number of flights/launches or an arbitrary time period, etc.). 

The answers generated can be substantially different due to the criteria used. 

3 ) Require all estimates to be created by a bonafide estimating department that has 
direct reporting to senior management, similar to SM&A organizations 

4 ) Recognize that cost control is just as important as cost estimates, and require valid 
cost impacts of all changes to be estimated, and not allow them to be labeled as 
“no cost” changes. All changes cost something! 

5 ) Require managers to identify and publicize all valid elements/components identified 
by qualified personnel even if those items are not admitted or baselined by the 
program and even if those elements cause funding distress. 

6) Require cost estimates to be submitted in future year dollars reflecting historical 
long term escalation rates - never in base year dollars. Base year dollars compared 
to then year dollars, adds to the illusion of cost growth even if there is not any. 

7 ) Require that the current development stage of the program be specified on the 
estimate. Providing project phase information is key to historical analysis integrity. 

8 ) Disenfranchise the risk reward system that provides strong incentives for 
underestimation. 

■ Specify procedures that will be implemented to cancel projects that exceed 
schedule and budget. 

■ Designate evaluation criteria that will hold project managers accountable for 
their management actions/decisions. 

■ Reward success for submitting realistic estimates and accomplishing them as 
advertised. 

■ Punish failure, swiftly and publicly 

■ Use fee incentives like cost plus (fixed fee) or cost plus (incentive fee) that 
penalizes the contractor for over-run - reward success, not the appearance of 

effort. 

■ Remove incentives for contractors to spend more time and money in order to 
increase profits. 

■ Define requirements and issue more firm fixed price contracts, and don’t make 
changes to them. Contractors will then start providing realistic proposals if they 
know they will have to stick with them. 

9) Remove the prevailing stigma that under-runs are unacceptable. In other words, 
reverse the policy metrics which mandate that all funding must be spent or that 
frugal spending is viewed as bad. If a project has any residual funding, the Monte 
Carlo concept, which allocates excess monies in the direction of over-runs on other 
projects, relies on underruns on some projects. As it now stands, NASA & 
congress are insisting that there “shall be no under-runs". 

We are gratified that NASA is becoming more committed to joint cost-schedule confidence assessment. 
Yet, we feel the needed course correction is not yet where it needs to be. We affirm our conviction that 
throughout all levels of the NASA hierarchy we all must constantly engage in self evaluation and 
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regularly ask ourselves the timely question: Are we meeting the intent of our former NASA 
Administrator? To wit: 


initiate a pattern of honest dealing between Program and Project Managers, Head 
Quarters, the Congress, and the White House, and . . . avoid the pattern of finger-pointing 
for cost overruns and schedule slips that have plagued the industry in the past.” 

Michael Griffin - NASA Administrator - 3/27/2006 


This can only be accomplished by 1) organizational frankness and individual honesty at every level of 
the budgeting process, 2) employing the use of sophisticated and intelligent assessment tools that result 
in estimate projections which history will validate as intuitive, timely, and accurate. To this end we 
submit for your consideration this paper and its recommendations. 


Page 96 


4/17/2009 


NASA 's Joint Confidence Level Paradox - A History of Denial 


2009 Cost Estimating Symposium 


Appendix 

In an attempt to aid future researchers on this subject, here is the project cost data used in the analysis. 
Do to the prevailing difficulty in collecting accurate data, it is anticipated that some of the cost growth is 
understated. This is caused by project renaming, re-baselining, scope cuts and project cancelations. 

For example the COSTR program was reported in a 2004 CBO analysis with an initial estimate of $22 1 
million and a current estimate of $326 million and a cost growth of 48%. However a 1992 analysis by 
the GAO references the initial estimate as being $400 million and a current estimate of $673 million, 
with a cost growth of 68%. Our analysis takes the CBO’s earliest estimate of $221 million and the 
GAO’s latest estimate of $673 million for a true cost growth of 204.5%. 

Others projects are possibly overstated due to apples and oranges comparisons - for example, comparing 
projects base year dollars to then year dollars. Although this is not a valid comparison, it is 
unfortunately the one often used in the court of public opinion when criticizing NASA’s estimates. 
Numerous sources were used in search of credible earliest available and latest available estimates. 

CBO - A Budgetary Analysis of NASA’s New Vision for Space Exploration September 2004 

GAO/NSLAD-93-97 NASA Program Costs 

GAO-04-642 Lack of Disciplined Cost-Estimating Processes Hinders Effective Program Management 
Various websites and news publications 

Overall it is felt that project true cost growth is much larger than generally reported or acknowledged. A 
IG report on the SOFIA project illustrates the point. 


Year of 
Projection 

Projected Budget for Initial 
Program Development 

Projected Date for 
Operational Readiness Review 

Note 

1997 

$ 265 million 

September 2001 

1 

2000 

301 million 

December 2002 

1 

2003 

373 million 

March 2005 

1 

2008 

840 million* 

August 2011 

2 


Note 1: Based on SOFIA Independent Cost. Schedule, and Management Review of October 2004. 

Note 2: Dollar projection based on the NASA FY 2009 Budget Estimate: date is the Limited Operational 
Capability milestone. 

* Does not include S3 5 million for formulation. 

Figure 72: - 3/27/09 NASA IG report on SOFIA project cost history - our analysis used the $373 million cost numbers. 
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NASA Historical Cost Growth - Used in Paper 

This is the data used in the paper. Numerous sources were used in search of credible earliest available 
and latest available estimates. All data comes from reputable sources, however errors probably exist, 
and some projects are still in development, so values may continue to evolve. See last data set for best 
available information. 


Table 4 



Earliest 

Latest 

Percent 

Program 

Available 

Available 

Change 

Advanced Communications Technology Satellite (ACTS) 

$ 

354.00 

$ 

656.00 

85.3% 

Advanced Health Management System Ph 1 (AHMS Phase 1) 

$ 

55.00 

$ 

55.00 

0.0% 

Advanced Solid Rocket Motor (ASRM) 

$ 

1,699.00 

$ 3,251.80 

91.4% 

Advanced X-ray Astrophysics Facility (AXAF) 

$ 

1,410.00 

$ 6,022.00 

327.1% 

Aeroassist Flight Experiment (AFE) 

$ 

159.00 

$ 

387.00 

143.4% 

Alternate Turbopumps (ATP) 

$ 

372.00 

$ 1,053.00 

183.1% 

Apollo 

$ 

7,000.00 

$25,400.00 

262.9% 

Aqua 

$ 

762.50 

$ 

1,006.00 

31.9% 

Cassini 

$ 

1,436.40 

$ 

1,375.90 

-4.2% 

Checkout and Launch Control System (CLCS) 
Cloud-Aerosol Lidar and Infrared Pathfinder Satellite 

$ 

175.00 

$ 

399.00 

128.0% 

Observations (CALIPSO) 

$ 

68.20 

$ 

133.90 

96.3% 

CloudSat Spacecraft 

$ 

80.20 

$ 

105.80 

31.9% 

Cockpit Avionics Upgrade (CAU) 

$ 

442.00 

$ 

454.00 

2.7% 

Collaborative Solar Terrestrial Research (COSTR) 

$ 

221.00 

$ 

673.00 

204.5% 

Comet Nucleus Tour 

$ 

69.10 

$ 

96.50 

39.7% 

Comet Rendezvous Asteroid Flyby (CRAF)/ Cassini (Mission 
to Saturn) 

$ 

3,593.00 

$ 

3,351.00 

-6.7% 

Cosmic Background Explorer 

$ 

97.50 

$ 

159.70 

63.8% 

Deep Space- 1 

$ 

73.30 

$ 

94.80 

29.3% 

Earth Observing System, Aura Satellite 

$ 

524.00 

$ 

763.00 

45.6% 

Earth Observing System, Terra Satellite 

$ 

1,078.70 

$ 

1,226.50 

13.7% 

Earth Observing-1 

$ 

72.00 

$ 

158.00 

119.4% 

Earth System Science Pathfinder 

$ 

145.10 

$ 

171.80 

18.4% 

Environmental Research Aircraft & Sensor Technology 

$ 

181.30 

$ 

173.00 

-4.6% 

Extreme Ultraviolet Explorer (EUVE) 

$ 

107.40 

$ 

322.00 

199.8% 

Far Ultraviolet Spectroscopic Explorer 

$ 

85.90 

$ 

120.40 

40.2% 

Fast Auroral Snapshot Explorer 

$ 

32.50 

$ 

42.90 

32.0% 

Flight Telerobotic Servicer (FTS) 

$ 

317.00 

$ 

485.00 

53.0% 

Fluids and Combustion Facility (FCF) 

$ 

118.90 

$ 

114.10 

-4.0% 

Fourier Transform Spectrometer 

$ 

317.00 

$ 

453.20 

43.0% 

Freedom (Space Station) 

$ 

25.12 

$ 

28.94 

15.2% 

Galaxy Evolution Explorer 

$ 

41.10 

$ 

87.10 

111.9% 

Galileo 

$ 

276.20 

$ 

902.30 

226.7% 

Galileo (Mission to Jupiter) 

$ 

455.00 

$ 

1,639.00 

260.2% 

Gamma Ray Observatory (GRO) 

$ 

183.80 

$ 

677.00 

268.3% 

Genesis Spacecraft 

$ 

126.10 

$ 

151.50 

20.1% 

Geospatial Operational Environmental Satellite l-M 

$ 

554.60 

$ 

1,241.00 

123.8% 

Geostationary Operational Environmental Satellite (GOES) 

$ 

691.00 

$ 

1,787.00 

158.6% 

Global Geospace Science (GGS) 

$ 

334.00 

$ 

649.00 

94.3% 

Gravity Probe B (GP-B) 

$ 

529.60 

$ 

709.30 

33.9% 

High Energy Transient Explorer-ll 

$ 

8.40 

$ 

23.50 

179.8% 

High Energy Solar Spectroscopic Imager 

$ 

39.50 

$ 

63.50 

60.8% 

Hubble Space Telescope (HST) 

$ 

435.00 

$ 

1,682.00 

286.7% 
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Ice Cloud and Land Elevation Satellite 

$ 

121.30 

$ 

177.00 

45.9% 

Imager for Aurora to Magnetopause Global Exploration 

$ 

83.60 

$ 

89.20 

6.7% 

International Gamma-Ray Astrophysics Lab (INTEGRAL) 

$ 

8.20 

$ 

11.90 

45.1% 

Jason- 1 

$ 

77.50 

$ 

87.80 

13.3% 

James Webb Space Telescope (JWST) 

$ 

900.00 

$ 4,900.00 

444.4% 

Land Remote Sensing Satellite (LANDSAT-D) 

$ 

260.10 

$ 

538.00 

106.8% 

Landsat-7 

$ 

445.80 

$ 

508.80 

14.1% 

Large Throat Main Combustion Chamber 

$ 

87.10 

$ 

76.90 

-11.7% 

Lunar Prospector 

$ 

56.20 

$ 

56.20 

0.0% 

Magellan (Mission to Venus) 

$ 

322.80 

$ 

856.00 

165.2% 

Mars Climate Orbiter 

$ 

183.60 

$ 

189.70 

3.3% 

Mars Exploration Rovers (MERs) 2003 

$ 

499.40 

$ 

767.00 

53.6% 

Mars Global Surveyor 

$ 

140.20 

$ 

130.70 

-6.8% 

Mars Observer (Mission to Mars) 

$ 

306.00 

$ 

994.00 

224.8% 

Mars Odyssey 

$ 

267.20 

$ 

366.10 

37.0% 

Mars Pathfinder - Total Cost Including Launch Vehicle 265.4M 

$ 

150.00 

$ 

174.20 

16.1% 

Mars Science Laboratory 

$ 

650.00 

$ 

2,300.00 

253.8% 

Mercury Program 

$ 

196.92 

$ 

384.00 

95.0% 

Microwave Anisotropy Probe 

$ 

88.30 

$ 

94.20 

6.7% 

Multifunction Electronics Display Subsystem 

$ 

201.70 

$ 

210.10 

4.2% 

NASA Scatterometer (NSCAT) 

$ 

100.40 

$ 

255.00 

154.0% 

Near Earth Asteroid Rendezvous Mission (NEAR) 

$ 

150.00 

$ 

124.90 

-16.7% 

New Millennium Program Earth Observing-1 (NMP-EO-1) 

$ 

111.70 

$ 

176.40 

57.9% 

Ocean Topography Experiment (TOPEX) 

$ 

438.00 

$ 

520.00 

18.7% 

Orbital Maneuvering Vehicle (OMV) 

$ 

371.00 

$ 

814.00 

119.4% 

Rosetta 

$ 

28.40 

$ 

40.10 

41.2% 

SeaWinds 

$ 

130.20 

$ 

148.80 

14.3% 

Second Tracking and Data Relay Satellite Ground Terminal 

$ 

341.40 

$ 

532.00 

55.8% 

Solar Radiation and Climate Experiment 

$ 

68.00 

$ 

74.50 

9.6% 

Solar Terrestrial Relations Observatory (STEREO) 

Space Infrared Telescope Facility (SIRTF) Renamed Spitzer 

$ 

150.00 

$ 

550.00 

266.7% 

Space Telescope 12-03 

$ 

472.00 

$ 

712.00 

50.8% 

Space Shuttle 

$ 

5,800.00 

$17,789.00 

206.7% 

Space Shuttle - With Reserves 

$ 

6,960.00 

$17,789.00 

155.6% 

Space Shuttle Endeavour 

$ 

2,100.00 

$ 

1,800.00 

-14.3% 

Space Station 

$ 

9,446.24 

$45,000.00 

376.4% 

Stardust Spacecraft 

$ 

117.80 

$ 

116.80 

-0.8% 

Stratospheric Observatory for Infrared Astronomy (SOFIA) 
Submillimeter Wave Astronomy Satellite/Transition Region and 

$ 

234.80 

$ 

373.00 

58.9% 

Coronal Explorer/Wide Field Infrared Explorer 

$ 

140.00 

$ 

212.70 

51.9% 

Terra 

$ 

1,309.10 

$ 

1,393.20 

6.4% 

Tethered Satellite Svstem (TSS) 

Thermosphere, Ionosphere, Mesosphere Energetics and 

$ 

40.70 

$ 

263.00 

546.2% 

Dynamics (TIMED) 

$ 

129.30 

$ 

176.20 

36.3% 

Topography Experiment 

$ 

321.30 

$ 

401.50 

25.0% 

Tracking and Data Relay Satellite replacement (TDRS-7) 

$ 

300.00 

$ 

532.00 

77.3% 

Tracking and Data Relay Satellite System Replenishment 

$ 

899.80 

$ 

803.10 

-10.7% 

Tracking and Data Relay Satellite-7 

$ 

269.00 

$ 

370.00 

37.5% 

Triana Spacecraft 

$ 

75.00 

$ 

96.90 

29.2% 

Tropical Rainfall Measuring Mission (TRMM) 

$ 

218.80 

$ 

468.00 

113.9% 

Ulysses (Mission to the Sun) 

$ 

196.00 

$ 

460.00 

134.7% 

Upper Atmosphere Research Satellite (UARS) 

$ 

575.30 

$ 

790.00 

37.3% 

X-33 

$ 

1,124.00 

$ 

1,789.70 

59.2% 

X-34 

$ 

171.00 

$ 

378.00 

121.1% 
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X-38 

X-43 Hyper-X 

X-ray Timing Explorer (XTE) 


$ 500.00 $ 1,500.00 200.0% 

$ 167.00 $ 227.00 35.9% 

$ 100.00 $ 373.00 273.0% 


Manned Space Flight Appears Riskier 

Manned space flight projects appear to experience on average substantially higher cost growth, with an 
average growth of 1 56% and a median growth of 1 28%. 


Table 5 


Program 

Initial 

Latest 

Available 

Percent 

Change 

Alt Turbopumps 

$ 372.00 

$ 1,053.00 

183% 

CLCS - Canceled 

$ 175.00 

$ 399.00 

128% 

Mercury Program 

$ 196.92 

$ 384.00 

95% 

Space Shuttle 

$ 5,800.00 

$17,789.00 

207% 

Space Station 

$ 9,446.24 

$45,000.00 

376% 

X-30 National Aero Space Plane - Canceled 

$ 3,100.00 

$10,000.00 

223% 

X-33 - Canceled 

$ 1,124.00 

$ 1,789.70 

59% 

X-34 - Canceled 

$ 171.00 

$ 378.00 

121% 

X-38 - Canceled 

$ 500.00 

$ 1,500.00 

200% 

X-43 Hyper-X - Canceled 

$ 167.00 

$ 227.00 

36% 

Advanced Solid Rocket Motor (ASRM) - Canceled 

$ 1,699.00 

$ 3,251.80 

91% 


Additional Cost & Schedule Growth Data Set #1 

The following cost and schedule growth data was compiled by Mr. Christian Smart of SAIC, and is 
included for additional reference information. This data was not obtained until after paper was 
completed, so none was used in our analysis. 

It is important to note that generally this data indicates much less cost growth, with a median cost 
growth of 24.9% and a average of 45.5%, and a median schedule growth of 17%, and an average 
schedule growth of 2 1 .9%. 

There are deltas between initial costs, and final costs on many of the projects. With some projects 
showing higher cost growth. The “correct” answer probably requires another scrub of the data 
compiling the lowest indicated estimate, and the highest indicated estimate for each project, which time 
did not allow. 




Initial 

Budget 

Final 

Budget 

Cost 

Initial 

Final 

Schedule 

Mission 


Growth 

% 

Schedule 

(Months) 

Schedule 

(Months) 

Growth 

% 

ACE 

$ 

141.1 

$ 108.5 

-23% 

61.7 

61.7 

0% 

Alt. Turbopump 

$ 

591.7 

$ 993.0 

68% 




ASRM 

$ 

1,506.7 

$3,251.8 

116% 




AURA 

$ 

660.5 

$ 713.8 

8% 

113.6 

133.4 

17% 

AXAF 

$ 

1,410.0 

$ 1,617.8 

15% 




CAL IPSO 

$ 

68.2 

$ 170.0 

149% 

51.0 

89.5 

75% 

Cassini 

$ 

1,436.4 

$ 1,375.9 

-4% 




CLCS 

$ 

175.0 

$ 390.1 

123% 




CloudSat 

$ 

80.2 

$ 144.0 

80% 

46.2 

85.2 

84% 

CONTOUR 

$ 

77.9 

$ 96.8 

24% 

39.6 

39.6 

0% 

Deep Space 1 

$ 

88.2 

$ 99.3 

13% 

33.5 

37.3 

11% 
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EO-1 

$ 

73.8 

$ 

158.1 

114% 

38.5 

57.5 

49% 

ERAST 

$ 

181.3 

$ 

173.0 

-5% 




ESSP 









(VCL/GRACE) 

$ 

145.1 

$ 

171.8 

18% 




FAST 

$ 

36.1 

$ 

42.9 

19% 

74.6 

88.5 

19% 

FUSE 

$ 

108.0 

$ 

120.4 

11% 

44.0 

51.9 

18% 

GALEX 

$ 

45.6 

$ 

72.0 

58% 

39.6 

59.7 

51% 

Genesis 

$ 

137.5 

$ 

162.9 

18% 

38.6 

45.9 

19% 

GGS (Wind/Polar) 

$ 

334.0 

$ 

458.1 

37% 




HESSI 

$ 

32.0 

$ 

46.6 

46% 

31.4 

50.9 

62% 

HETE-II 

$ 

8.4 

$ 

14.4 

71% 

36.5 

49.0 

34% 

ICESat 

$ 

121.3 

$ 

177.0 

46% 

60.9 

73.4 

21% 

IMAGE 

$ 

83.6 

$ 

89.2 

7% 

46.7 

48.5 

4% 

LTMCC 

$ 

87.1 

$ 

76.9 

-12% 




Lunar Prospector 

$ 

56.2 

$ 

56.6 

1% 

30.5 

33.7 

10% 

MAP 

$ 

88.3 

$ 

94.2 

7% 

55.8 

63.9 

15% 

Mars Odyssey 

$ 

267.2 

$ 

366.1 

37% 




Mars Pathfinder 

$ 

174.2 

$ 

174.2 

0% 

37.5 

37.6 

0% 

MCO 

$ 

183.6 

$ 

189.7 

3% 

38.6 

38.9 

1% 

MEDS 

$ 

201.7 

$ 

210.1 

4% 




MER 

$ 

533.1 

$ 

618.9 

16% 

34.1 

34.5 

1% 

MGS 

$ 

140.2 

$ 

130.7 

-7% 

33.7 

33.7 

0% 

NEAR 

$ 

150.1 

$ 

124.9 

-17% 

29.0 

29.0 

0% 

OMV 

$ 

466.0 

$ 

766.5 

64% 




SIRTF 

$ 

472.7 

$ 

644.2 

36% 

67.0 

88.0 

31% 

SLWT 

$ 

172.5 

$ 

129.0 

-25% 




SOFIA 

$ 

239.4 

$ 

373.0 

56% 




SORCE 

$ 

68.0 

$ 

74.5 

10% 

48.0 

54.9 

14% 

STARDUST 

$ 

117.8 

$ 

116.8 

-1% 

38.8 

38.8 

0% 

TDRSS replen 

$ 

899.8 

$ 

803.1 

-11% 




Terra 

$ 

1,078.7 

$ 

1,226.5 

14% 




TIMED 

$ 

129.2 

$ 

162.3 

26% 

42.6 

67.2 

58% 

Triana 

$ 

75.0 

$ 

96.9 

29% 




TRMM 

$ 

253.1 

$ 

246.0 

-3% 

83.2 

87.1 

5% 

X-33 

$ 

1,075.2 

$1,218.9 

13% 




X-34 

$ 

70.0 

$ 

205.2 

193% 




X-38 

$ 

80.0 

$ 

94.0 

18% 




X-43 

$ 

167.0 

$ 

227.0 

36% 




XTE 

$ 

109.2 

$ 

194.2 

78% 




SWAS 

$ 

47.3 

$ 

78.9 

67% 

73.0 

116.4 

59% 

Landsat-7 

$ 

387.1 

$ 

449.1 

16% 

75.1 

79.6 

6% 

Aqua 

$ 

946.7 

$ 

884.1 

-7% 

89.3 

106.6 

19% 

Gravity Probe B 

$ 

351.0 

$ 

499.9 

42% 

85.2 

128.5 

51% 

TRACE 

$ 

35.6 

$ 

40.3 

13% 

48.2 

52.3 

8% 

WIRE 

$ 

39.7 

$ 

50.7 

28% 

43.6 

50.8 

17% 

ICESAT 

$ 

121.3 

$ 

177.0 

46% 

60.9 

73.4 

21% 

GRACE 

$ 

79.3 

$ 

88.4 

11% 

50.7 

60.4 

19% 

Deep Impact 

$ 

194.1 

$ 

252.0 

30% 

51.2 

63.3 

24% 

Messenger 

$ 

191.5 

$ 

288.7 

51% 

51.7 

56.9 

10% 

SWIFT 

$ 

102.4 

$ 

164.9 

61% 

44.6 

59.5 

33% 

STEREO 

$ 

267.6 

$ 

351.3 

31% 

55.8 

64.6 

16% 
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MRO 

$ 

334.4 

$ 

359.0 

7% 

46.7 

47.0 

1% 

THEMIS 

$ 

102.3 

$ 

107.6 

5% 

42.2 

43.5 

3% 

Tether 

$ 

28.3 

$ 

115.7 

309% 




HST-OTA 

$ 

115.8 

$ 

561.7 

385% 




OMV 

$ 

236.0 

$ 

523.1 

122% 




ET 

$ 

349.6 

$ 

961.7 

175% 




SSME 

$ 

1,267.1 

$3,051.5 

141% 




SRM 

$ 

338.6 

$ 

706.7 

109% 




AIM 

$ 

61.1 

$ 

81.2 

33% 

43.8 

49.9 

14% 

Deep Impact 

$ 

194.1 

$ 

252.0 

30% 

51.2 

63.3 

24% 

ST-5 

$ 

26.3 

$ 

24.7 

-6% 

56.0 

78.0 

39% 


Christian’s Sources 
Shaffer 
Rutkowski 
REDSTAR 121- 
4955 <-FY92$ 

Freaner-Bitten 

Additional Cost & Schedule Growth Data Set #2 

The following cost and schedule growth data was compiled by Debra L. Emmons, Robert E. Bitten, 
Claude W. Freaner, for their paper titled “Using Historical NASA Cost and Schedule Growth to Set 
Future Program and Project Reserve Guidelines” and is included for additional reference information. 
This data was not obtained until after paper was completed, so none was used in our analysis. 


Mission 

Initial Cost 
Estimate 

Final Cost 

% Cost 
Growth 

Initial 

Schedule 

Months 

Final 

Schedule 

Months 

% 

Schedule 

Growth 

NEAR 

$ 150.1 

$ 

124.9 

-16.8% 

29.0 

29.0 

0.0% 

Lunar Prospector $ 56.2 

$ 

56.6 

0.7% 

30.5 

33.7 

10.7% 

Genesis 

$ 137.5 

$ 

162.9 

18.5% 

38.6 

45.9 

18.9% 

Messenaer 

$ 191.5 

$ 

288.7 

50.8% 

51.7 

56.9 

10.0% 

Mars Pathfinder 

$ 174.2 

$ 

174.2 

0.0% 

37.5 

37.6 

0.3% 

Stardust 

$ 127.4 

$ 

126.4 

-0.8% 

38.8 

38.8 

0.0% 

Contour 

$ 77.9 

$ 

96.8 

24.3% 

39.6 

39.6 

0.2% 

DeeD Impact 

$ 194.1 

$ 

252.0 

29.8% 

51.2 

63.3 

23.6% 

MGS 

$ 140.2 

$ 

130.7 

-6.8% 

33.7 

33.7 

0.0% 

MCO/MPL 

$ 183.6 

$ 

189.7 

3.3% 

38.6 

38.9 

0.9% 

MER 

$ 533.1 

$ 

618.9 

16.1% 

34.1 

34.5 

1.1% 

MRO 

$ 394.4 

$ 

450.0 

14.1% 

46.7 

46.7 

0.0% 

FAST 

$ 36.1 

$ 

42.9 

18.9% 

74.6 

88.5 

18.6% 

SWAS 

$ 47.3 

$ 

78.9 

66.8% 

73.0 

116.4 

59.4% 

TRACE 

$ 35.6 

$ 

40.3 

13.2% 

48.2 

52.3 

8.4% 

WIRE 

$ 39.7 

$ 

50.7 

27.7% 

43.6 

50.8 

16.5% 

ACE 

$ 152.0 

$ 

119.4 

-21.4% 

61.7 

61.7 

0.0% 

FUSE 

$ 131.3 

$ 

143.7 

9.4% 

44.0 

51.9 

17.8% 

IMAGE 

$ 83.6 

$ 

89.2 

6.7% 

46.7 

48.5 

3.9% 

MAP 

$ 88.3 

$ 

94.2 

6.7% 

55.8 

63.9 

14.4% 

HESSI 

$ 32.0 

$ 

46.6 

45.6% 

31.4 

50.9 

61.9% 

GALEX 

$ 45.6 

$ 

72.0 

57.9% 

39.6 

59.7 

50.8% 

SWIFT 

$ 102.4 

$ 

164.9 

61.0% 

44.6 

59.5 

33.3% 
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GRACE 

$ 79.3 

$ 88.4 

11.5% 

50.7 

60.4 

19.0% 

CLOUDSAT 

$ 80.2 

$ 144.0 

79.6% 

46.2 

85.2 

84.2% 

CALIPSO 

$ 68.2 

$ 170.3 

149.7% 

51.0 

89.5 

75.4% 

DS-1 

$ 88.2 

$99.3 

12.6% 

33.5 

37.3 

11.5% 

EO-1 

$ 73.8 

$ 158.1 

114.2% 

38.5 

57.5 

49.3% 

SIRTF 

$ 472.7 

$ 644.2 

36.3% 

67.0 

88.0 

31.5% 

STEREO 

$ 267.6 

$ 351.3 

31.3% 

55.8 

64.6 

15.6% 

EOS-Aaua 

$ 946.7 

$ 884.1 

-6.6% 

89.3 

106.6 

19.4% 

EOS-Aura 

$ 660.5 

$ 713.8 

8.1% 

113.6 

133.4 

17.4% 

Landsat-7 

$ 387.1 

$ 449.1 

16.0% 

75.1 

79.6 

6.0% 

TRMM 

$ 253.1 

$ 246.0 

-2.8% 

83.2 

87.1 

4.7% 

TIMED 

$ 129.2 

$ 162.3 

25.6% 

43.6 

67.2 

53.9% 

Gravity Probe B 

$351.0 

$ 499.9 

42.4% 

85.2 

127.8 

50.0% 

THEMIS 

$ 102.3 

$ 107.6 

5.2% 

42.2 

43.5 

3.1% 

HETE-II 

$8.4 

$14.4 

71.4% 

36.5 

49.0 

34.2% 

SORCE 

$ 68.0 

$74.5 

9.6% 

48.0 

54.9 

14.4% 

ICESAT 

$ 121.3 

$ 177.0 

45.9% 

60.9 

73.4 

20.6% 


Additional Cost & Schedule Growth Data Set #3 

The following cost and schedule growth data was compiled by from Kelli McCoy’s 2008 NASA Cost 
Symposium paper, and is included for additional reference information. This data was not used in our 
analysis. 


Theme 

Mission 

Original 

Duration 

Final 

Duration 

Slip % 

SRR 

year 

Launch 

Heliophysics 

ACE 

57 

57 

0% 

1992 

1997 

Earth Sci 

ACRIMSAT 

43 

45 

5% 

1996 

1999 


ACTS 

48 

98 

104% 

1985 

1993 

Heliophysics 

AIM 

40 

47 

18% 

2003 

2007 

Earth Sci 

Aquarius 

50 

69 

38% 

2004 

2010 

Earth Sci 

Aura (Chem-1) or Chemistry 

41 

60 

46% 

1999 

2004 

Earth Sci 

CALIPSO 

38 

75 

97% 

2000 

2006 

Planetary 

Cassini 

92 

110 

20% 

1988 

1997 

ASO 

Chandra 

69 

79 

14% 

1992 

1999 

ASO 

CHIPSAT 

30 

40 

33% 

1999 

2003 

Heliophysics 

CINDI 

41 

95 

132% 

2000 

2008 

Earth Sci 

CloudSat 

36 

74 

106% 

2000 

2006 


Cluster 

75 

81 

8% 

1989 

1996 

Heliophysics 

Cluster-2 (Salsa & Samba) 

75 

130 

73% 

1989 

2000 

ASO 

COBE 

68 

88 

29% 

1982 

1989 

Planetary 

CONTOUR 

38 

38 

0% 

1999 

2002 

Earth Sci 

CRRES 

38 

86 

126% 

1983 

1990 


DART 

31 

43 

39% 

2001 

2005 

Planetary 

DAWN 

37 

53 

43% 

2003 

2007 

Planetary 

Deep Impact 

44 

56 

27% 

2000 

2005 

Planetary 

Deep Space 1 

40 

43 

8% 

1995 

1998 

Earth Sci 

EO-1 

33 

53 

61% 

1996 

2000 

ASO 

EUVE 

48 

58 

21% 

1987 

1992 

Heliophysics 

FAST 

44 

67 

52% 

1991 

1996 

ASO 

FUSE 

100 

107 

7% 

1990 

1999 
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ASO 

GALEX 

37 

56 

51% 

1998 

2003 

Planetary 

Galileo Orbiter 

43 

136 

216% 

1978 

1989 

Planetary 

Galileo Probe 

43 

136 

216% 

1978 

1989 

Earth Sci 

Genesis 

34 

41 

21% 

1998 

2001 

Heliophysics 

Geotail 

58 

58 

0% 

1987 

1992 

ASO 

GLAST 

66 

93 

41% 

2000 

2008 

Earth Sci 

Glory 

88 

106 

20% 

2000 

2009 

Earth Sci 

GOES 1 

45 

102 

127% 

1985 

1994 

Earth Sci 

GOES J 

53 

115 

117% 

1985 

1995 

Earth Sci 

GOESK 

57 

138 

142% 

1985 

1997 

Earth Sci 

GOES L 

65 

175 

169% 

1985 

2000 

Earth Sci 

GOESM 

74 

189 

155% 

1985 

2001 

Earth Sci 

GOES N 

41 

96 

134% 

1998 

2006 

Earth Sci 

GOESO 

47 

126 

168% 

1998 

2008 

Earth Sci 

GOESP 

83 

137 

65% 

1998 

2009 

ASO 

GPB (Gravity Probe B) 

74 

126 

70% 

1993 

2004 

Earth Sci 

GPM 

113 

149 

32% 

2002 

2014 

Earth Sci 

GRACE 

GRO - (Compton Gamma Ray 

42 

51 

21% 

1997 

2002 

ASO 

Ob) 

62 

127 

105% 

1980 

1991 

ASO 

HST OTA 

70 

146 

109% 

1978 

1990 

ASO 

HST SI 

64 

140 

119% 

1978 

1990 

ASO 

HST SSM 

70 

146 

109% 

1978 

1990 

Heliophysics 

IBEX 

33 

36 

9% 

2005 

2008 

Earth Sci 

ICESAT 

43 

55 

28% 

1998 

2003 

Heliophysics 

IMAGE 

42 

44 

5% 

1996 

2000 

Earth Sci 

JASON 1 

70 

94 

34% 

1994 

2001 

Planetary 

JUNO 

51 

51 

0% 

2007 

2011 

ASO 

JWST 

92 

114 

24% 

2003 

2013 

ASO 

Kepler 

48 

64 

33% 

2003 

2009 

Earth Sci 

Landsat 7 

55 

66 

20% 

1993 

1999 


LCROSS 

27 

28 

4% 

2006 

2008 

Earth Sci 

LDCM 

38 

38 

0% 

2008 

2011 

ASO 

LISA 

55 

120 

118% 

2007 

2017 

Planetary 

MAGELLAN 

61 

73 

20% 

1983 

1989 

ASO 

MAP or WMAP 

54 

61 

13% 

1996 

2001 

Planetary 

MARS Climate Orbiter - MCO 

45 

45 

0% 

1995 

1998 

Planetary 

Mars Observer 

100 

126 

26% 

1982 

1992 

Planetary 

Mars Odyssey 01 

33 

34 

3% 

1998 

2001 

Planetary 

Mars Pathfinder 

49 

49 

0% 

1992 

1996 

Planetary 

Mars Polar Lander (MPL) 

45 

46 

2% 

1995 

1999 

Planetary 

MER-A or MER03 - SPIRIT) 

35 

35 

0% 

2000 

2003 

Planetary 

MER-B (Opportunity) 

35 

36 

3% 

2000 

2003 

Planetary 

MESSENGER 

46 

51 

11% 

2000 

2004 

Planetary 

MGS -Mars Global Surveyor 

31 

31 

0% 

1994 

1996 

Planetary 

MMM (M3) on Chandrayaan-1 

31 

35 

13% 

2005 

2008 

Heliophysics 

MMS 

85 

85 

0% 

2007 

2014 

Planetary 

MRO 

43 

43 

0% 

2002 

2005 

Planetary 

MSL 

45 

45 

0% 

2005 

2009 


Page 104 


4/17/2009 


NASA 's Joint ConGdence Level Paradox - A History of Denial 


2009 Cost Estimating Symposium 


Planetary 

New Horizons 

44 

44 

0% 

2002 

2006 

Earth Sci 

NPP 

79 

123 

56% 

2000 

2010 

Earth Sci 

OCO 

52 

67 

29% 

2003 

2009 

Earth Sci 

OSTM 

72 

74 

3% 

2002 

2008 

Planetary 

Phoenix/Scout 7 

41 

41 

0% 

2004 

2007 

Earth Sci 

QUICKSCAT 

14 

21 

50% 

1997 

1999 

Heliophysics 

RHESSI was HESSI 

31 

53 

71% 

1997 

2002 

Heliophysics 

Sampex 

37 

38 

3% 

1989 

1992 

Heliophysics 

SDO 

52 

68 

31% 

2003 

2008 

Heliophysics 

SET-1 

39 

89 

128% 

2003 

2011 

ASO 

SIM 

73 

191 

162% 

2003 

2019 

ASO 

SIRTF or Spitzer 

55 

78 

42% 

1997 

2003 

Heliophysics 

SNOE 

23 

35 

52% 

1995 

1998 

Heliophysics 

SoHO 

63 

72 

14% 

1989 

1995 

Heliophysics 

Solar B or HINODE 

58 

83 

43% 

1999 

2006 

Earth Sci 

SORCE 

32 

44 

38% 

1999 

2003 

Heliophysics 

ST-5 

48 

70 

46% 

2000 

2006 

ASO 

ST-7 

47 

89 

89% 

2002 

2009 

Heliophysics 

ST-8 

37 

32 

-14% 

2006 

2008 

Planetary 

STARDUST 

35 

35 

0% 

1996 

1999 

Heliophysics 

STEREO 

49 

77 

57% 

2000 

2006 

ASO 

SWAS 

57 

102 

79% 

1990 

1998 

ASO 

SWIFT 

41 

54 

32% 

2000 

2004 


TDRS-H 

41 

52 

27% 

1996 

2000 


TDRS-I 

47 

73 

55% 

1996 

2002 


TDRS-J 

53 

82 

55% 

1996 

2002 

Earth Sci 

Terriers 

31 

56 

81% 

1994 

1999 

Heliophysics 

THEMIS 

37 

43 

16% 

2003 

2007 

Heliophysics 

TIMED 

63 

86 

37% 

1994 

2001 

Earth Sci 

Topex/Poseidon 

105 

151 

44% 

1980 

1992 

Heliophysics 

TRACE 

26 

32 

23% 

1995 

1998 

Earth Sci 

TRMM 

72 

81 

13% 

1991 

1997 

Earth Sci 

UARS 

73 

95 

30% 

1983 

1991 

Heliophysics 

Ulysses 

40 

132 

230% 

1979 

1990 

Heliophysics 

WIND 

48 

71 

48% 

1988 

1994 

ASO 

WIRE 

52 

57 

10% 

1994 

1999 

ASO 

WISE 

42 

59 

40% 

2004 

2009 

ASO 

XTE or RXTE 

53 

48 

-9% 

1991 

1995 


Compiled Cost & Schedule Growth Data Set 

The following cost and schedule growth data is a combined list of the earliest available and latest 
available data for 188 projects. Some of the names are the same, but supplementary data led us to 
believe they were separate projects. In fact they may not be, renaming, rebaselineing, and whitewashing 
make this type of data mining and analysis very difficult. All data comes from reputable sources, 
however errors probably exist, and some projects are still in development, so values may continue to 
evolve. 
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It shows an average cost growth of 98.2%, a median cost growth of 53.3%, and average schedule growth 
of 56.8%, and a median schedule growth of 34.9%. This is abysmal to say the least and exceeds many of 
the recently published papers values. This data was not obtained until after paper was completed, so 
none was used in our analysis, but is included in an attempt to aide future researchers. 


Theme 

Name 

Initial 

Latest 

Available 

Percent 

change 

Initial 

Schedule 

Final 

Schedule 

Percent 

Change 

Heliophysics 

ACE 

$ 141.10 

$ 108.50 

-23.1% 

57 

62 

8.2% 

Earth Sci 

ACRIMSAT 




43 

45 

4.7% 


ACTS 

$ 354.00 

$ 656.00 

85.3% 

48 

98 

104.2% 


AFE 

$ 159.00 

$ 387.00 

143.4% 

Canceled 




AHMS 

$ 55.00 

$ 55.00 

0.0% 




Heliophysics 

AIM 

$61.10 

$81.20 

32.9% 

40 

50 

24.7% 


Apollo 

$7,000.00 

$ 25,400.00 

262.9% 




Earth Sci 

Aqua 

$ 762.50 

$1,006.00 

31.9% 

89 

107 

19.4% 

Earth Sci 

Aquarius 




50 

69 

38.0% 

Manned 

ASRM 

$1,506.70 

$3,251.80 

115.8% 

Canceled 



Manned 

ATP 

$ 372.00 

$1,053.00 

183.1% 





AURA 

Aura (Chem-1) or 

$ 524.00 

$ 763.00 

45.6% 

114 

133 

17.4% 

Earth Sci 

Chemistry 

AXAF 

$1,410.00 

$ 6,022.00 

327.1% 

41 

60 

46.3% 

Heliophysics 

BARREL 





54 


Earth Sci 

CALIPSO 

$ 68.20 

$ 170.3 

149.7% 

38 

89 

135.4% 

Planetary 

Cassini 

$1,436.40 

$ 1,375.90 

-4.2% 

92 

110 

19.6% 

Manned 

CAU 

$ 442.00 

$ 454.00 

2.7% 




ASO 

Chandra 




69 

79 

14.5% 

ASO 

CHIPSAT 




30 

40 

33.3% 

Heliophysics 

CINDI 




41 

95 

131.7% 

Manned 

CLCS 

Clementine 

$ 175.00 

$ 399.00 

128.0% 

Canceled 

19 


Earth Sci 

CloudSat 

$ 80.20 

$ 144.00 

79.6% 

36 

85 

136.6% 


Cluster 

Cluster-2 (Rumba & 




75 

81 

8.0% 

Heliophysics 

Tengo) 

Cluster-2 (Salsa & 





131 


Heliophysics 

Samba) 




75 

130 

73.3% 

ASO 

COBE 




68 

88 

29.4% 

Planetary 

CONTOUR 
Cosmic Background 

$69.10 

$ 96.80 

40.1% 

38 

40 

4.2% 


Explorer 

$ 97.50 

$ 159.70 

63.8% 





COSTR 

$221.00 

$ 673.00 

204.5% 





CRAF 

$3,593.00 

$3,351.00 

-6.7% 

Canceled/ Development 

Earth Sci 

CRRES 




38 

86 

126.3% 


DART 




31 

43 

38.7% 

Planetary 

DAWN 

202.8 

287.1 

41.6% 

37 

53 

43.2% 


Deep Impact 

$ 194.10 

$ 252.00 

29.8% 

44 

63 

43.9% 

Planetary 

Deep Space 1 

$ 73.30 

$ 99.30 

35.5% 

33 

43 

28.5% 


Earth Sci 


DSMS 

EO-1 

ERAST 

ESSP 


$ 72.00 
$ 181.30 
$ 145.10 


$ 158.1 
$ 173.00 
$ 171.80 


33 


36 

58 


74.2% 
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ET 

$ 349.60 

$961.70 

175.1% 




ASO 

EUVE 

$ 107.40 

$ 322.00 

199.8% 

48 

58 

20.8% 

Heliophysics 

FAST 

$ 32.50 

$ 42.90 

32.0% 

44 

89 

101.1% 


FCF 

$ 118.90 

$ 114.10 

-4.0% 





FTS 

$ 317.00 

$ 485.00 

53.0% 

Canceled 




FTS 

$ 317.00 

$ 453.20 

43.0% 




ASO 

FUSE 

$ 85.90 

$ 143.7 

67.3% 

44 

107 

143.2% 

ASO 

GALEX 

$41.10 

$87.10 

111.9% 

37 

60 

61.4% 

Planetary 

Galileo Orbiter 




43 

136 

216.3% 

Planetary 

Galileo Probe 




43 

136 

216.3% 

Planetary 

Galilleo 

$ 276.20 

$ 1,639.00 

493.4% 




Earth Sci 

Genesis 

$ 126.10 

$ 162.90 

29.2% 

34 

46 

34.9% 

Heliophysics 

Geospace RBSP 





44 


Heliophysics 

Geotail 




58 

58 

0.0% 


GGS 

$ 334.00 

$ 649.00 

94.3% 




ASO 

GLAST 




66 

93 

40.9% 

Earth Sci 

Glory 




88 

106 

20.5% 


GOES 

$ 554.60 

$ 1,241.00 

123.8% 





GOES 

$691.00 

$ 1,787.00 

158.6% 




Earth Sci 

GOES 1 




45 

102 

126.7% 

Earth Sci 

GOES J 




53 

115 

117.0% 

Earth Sci 

GOESK 




57 

138 

142.1% 

Earth Sci 

GOES L 




65 

175 

169.2% 

Earth Sci 

GOESM 




74 

189 

155.4% 

Earth Sci 

GOES N 




41 

96 

134.1% 

Earth Sci 

GOESO 




47 

126 

168.1% 

Earth Sci 

GOESP 




83 

137 

65.1% 

ASO 

GP-B 

$351.00 

$ 709.30 

102.1% 

74 

128 

72.7% 

Earth Sci 

GPM 




113 

149 

31.9% 

Earth Sci 

GRACE 

$ 79.30 

$ 88.40 

11.5% 

42 

60 

43.8% 

Planetary 

GRAIL 




8 

44 

450.0% 


GRO 

$ 183.80 

$ 677.00 

268.3% 





GRO - (Compton 







ASO 

Gamma Ray Ob) 




62 

127 

104.8% 


HESSI 

$ 32.00 

$ 63.50 

98.4% 

31 

51 

61.9% 

ASO 

HETE 

$8.40 

$ 23.50 

179.8% 

37 

49 

34.2% 


HST 

$ 435.00 

$ 1,682.00 

286.7% 




ASO 

HST SI 




64 

140 

118.8% 

ASO 

HST SSM 




70 

146 

108.6% 

ASO 

HST-OTA 

$ 115.80 

$561.70 

385.1% 

70 

146 

108.6% 

Heliophysics 

IBEX 




33 

36 

9.1% 

Earth Sci 

ICESAT 

$ 121.30 

$ 177.00 

45.9% 

43 

73 

70.7% 

Heliophysics 

IMAGE 

$ 83.60 

$ 89.20 

6.7% 

42 

49 

15.5% 

ASO 

INTEGRAL 

$8.20 

$ 11.90 

45.1% 




Earth Sci 

JASON 

$ 77.50 

$ 87.80 

13.3% 

70 

94 

34.3% 

Planetary 

JUNO 




51 

51 

0.0% 

ASO 

JWST 

$ 900.00 

$ 4,900.00 

444.4% 

92 

114 

23.9% 

ASO 

Kepler 




48 

64 

33.3% 


LADEE 





38 


Earth Sci 

Landsat 7 

$387.10 

$ 508.80 

31.4% 

55 

80 

44.7% 


LANDSAT-D 

$260.10 

$ 538.00 

106.8% 
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LCROSS 




27 

28 

3.7% 

Earth Sci 

LDCM 




38 

38 

0.0% 

ASO 

LISA 




55 

120 

118.2% 

SSE 

LRO 




38 

39 

2.6% 

Manned 

LTMCC 

$87.10 

$ 76.90 

-11.7% 




Planetary 

Lunar 

$ 56.20 

$ 56.60 

0.7% 

30 

34 

10.7% 

Planetary 

MAGELLAN 

$ 322.80 

$ 856.00 

165.2% 

61 

73 

19.7% 


MAP 

• $88.30 

$ 94.20 

6.7% 

56 

64 

14.4% 

ASO 

MAP or WMAP 




54 

61 

13.0% 

Planetary 

MPL 




45 

46 

2.2% 

Planetary 

MCO 

$ 183.60 

$ 189.70 

3.3% 

39 

45 

16.6% 


MEDS 

$201.70 

$210.10 

4.2% 




Planetary 

MER 

$ 499.40 

$ 767.00 

53.6% 

34 

34 

1.1% 


MER-A or MER03 - 







Planetary 

SPIRIT) 




35 

35 

0.0% 

Planetary 

MER-B (Opportunity) 




35 

36 

2.9% 

Manned 

MERCURY 

$ 196.92 

$ 384.00 

95.0% 




Planetary 

Messenger 

$ 191.50 

$ 288.70 

50.8% 

46 

57 

23.7% 

Planetary 

MGS 

$ 140.20 

$ 130.70 

-6.8% 

31 

34 

8.7% 

Planetary 

MMM 




31 

35 

12.9% 

Heliophysics 

MMS 




85 

85 

0.0% 

Planetary 

MRO 

$ 334.40 

$ 450.0 

34.6% 

43 

47 

9.4% 

Planetary 

MSL 

$ 650.00 

$ 2,300.00 

253.8% 

45 



Planetary 

NEAR 

$ 150.00 

$ 124.90 

-16.7% 

29 

29 

0.0% 

Planetary 

New Horizons 




44 

44 

0.0% 


NMP 

$ 111.70 

$ 176.40 

57.9% 




Earth Sci 

NPP 




79 

123 

55.7% 


NSCAT 

$ 100.40 

$ 255.00 

154.0% 




ASO 

NuStar 





37 


Planetary 

Mars Observer 

$ 306.00 

$ 994.00 

224.8% 

100 

126 

26.0% 

Earth Sci 

OCO 




52 

67 

28.8% 

Planetary 

ODYSSEY 

$ 267.20 

$366.10 

37.0% 

33 

34 

3.0% 

Manned 

OMV 

$ 236.00 

$814.00 

244.9% 

Canceled 



Earth Sci 

OSTM 




72 

74 

2.8% 

Planetary 

PATHFINDER 

$ 150.00 

$ 174.20 

16.1% 

38 

49 

30.6% 

Planetary 

Phoenix/Scout 7 




41 

41 

0.0% 

Earth Sci 

QUICKSCAT 




14 

21 

50.0% 

Heliophysics 

RHESSI was HESSI 




31 

53 

71.0% 


Rosetta 

$ 28.40 

$40.10 

41.2% 




Heliophysics 

Sampex 




37 

38 

2.7% 

Heliophysics 

SDO 




52 

68 

30.8% 


SeaWinds 

$ 130.20 

$ 148.80 

14.3% 




Heliophysics 

SET-1 




39 

89 

128.2% 

ASO 

SIM 




73 

191 

161.6% 

Manned 

SLWT 

$ 172.50 

$ 129.00 

-25.2% 




Heliophysics 

SNOE 




23 

35 

52.2% 

ASO 

SOFIA 

$ 234.80 

$ 840.00 

257.8% 

48 

207 

331.3% 

Heliophysics 

SoHO 




63 

72 

14.3% 


Solar Orbiter 





84 


Heliophysics 

Solar B or HINODE 

$ 99.30 

$ 80.40 

-19.0% 

58 

83 

43.1% 

Earth Sci 

SORCE 

$ 68.00 

$ 74.50 

9.6% 

32 

55 

71.6% 
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Manned 

Shuttle 

$5,800.00 

$ 17,789.00 

206.7% 




Manned 

Shuttle - With Reserves 

$6,960.00 

$ 17,789.00 

155.6% 




Manned 

Shuttle - Endeavor 

$2,100.00 

$ 1,800.00 

-14.3% 




Manned 

SSME 

$1,267.10 

$3,051.50 

140.8% 




Manned 

Space Station 

$9,446.24 

$ 45,000.00 

376.4% 




ASO 

SIRTF or Spitzer 

$ 472.00 

$ 712.00 

50.8% 

55 

88 

60.1% 

Manned 

SRM 

$ 338.60 

$ 706.70 

108.7% 




Manned 

SS 

$25.12 

$ 28.94 

15.2% 




Heliophysics 

ST-5 

$ 26.30 

48.7 

85.2% 

48 

78 

62.5% 

Heliophysics 

ST-6 





56 


ASO 

ST-7 




47 

89 

89.4% 

Heliophysics 

ST-8 




37 

32 

-13.5% 

CT 

ST-9 




42 

42 

0.0% 

Planetary 

STARDUST 

$ 117.80 

$ 126.4 

7.3% 

35 

39 

10.9% 


STDRS 

$341.40 

$ 532.00 

55.8% 




Heliophysics 

STEREO 

$ 150.00 

$ 550.00 

266.7% 

49 

77 

57.1% 

ASO 

SWAS 

$ 47.30 

$ 78.90 

66.8% 

57 

116 

104.2% 


SWASTR 

$ 140.00 

$212.70 

51.9% 




ASO 

SWIFT 

$ 102.40 

$ 164.90 

61.0% 

41 

60 

45.1% 


TDRS7 

$ 269.00 

$ 532.00 

97.8% 





TDRS-H 




41 

52 

26.8% 


TDRS-I 




47 

73 

55.3% 


TDRS-J 




53 

82 

54.7% 


TE 

$321.30 

$401.50 

25.0% 




Earth Sci 

TERRA 

$1,078.70 

$ 1,393.20 

29.2% 




Earth Sci 

Terriers 




31 

56 

80.6% 


Tether 

$ 28.30 

$ 115.70 

308.8% 




Heliophysics 

THEMIS 

$ 102.30 

$ 107.60 

5.2% 

37 

44 

17.6% 

Heliophysics 

TIMED 

$ 129.20 

$ 176.20 

36.4% 

44 

86 

97.1% 


TOPEX 

$ 438.00 

$ 520.00 

18.7% 




Earth Sci 

Topex/Poseidon 




105 

151 

43.8% 

Heliophysics 

TRACE 

$ 35.60 

$ 40.30 

13.2% 

26 

52 

101.2% 


TRDS 

$ 899.80 

$ 803.10 

-10.7% 





Triana Spacecraft 

$ 75.00 

$ 96.90 

29.2% 




Earth Sci 

TRMM 

$218.80 

$ 468.00 

113.9% 

72 

87 

21.0% 


TSS 

$ 40.70 

$ 263.00 

546.2% 




Earth Sci 

UARS 

$ 575.30 

$ 790.00 

37.3% 

73 

95 

30.1% 

Heliophysics 

Ulysses 

$ 196.00 

$ 460.00 

134.7% 

40 

132 

230.0% 

Heliophysics 

WIND 




48 

71 

47.9% 

ASO 

WIRE 

$ 39.70 

$ 50.70 

27.7% 

44 

57 

30.7% 

ASO 

WISE 




42 

59 

40.5% 

Manned 

X-30 

$3,100.00 

$ 10,000.00 

222.6% 

Canceled 



Manned 

X-33 - Canceled 

$1,075.20 

$ 1,789.70 

66.5% 

Canceled 



Manned 

X-34 - Canceled 

$ 70.00 

$ 378.00 

440.0% 

Canceled 



Manned 

X-38 - Canceled 

$ 500.00 

$ 1,500.00 

200.0% 

Canceled 



Manned 

X-43 Hyper-X Canceled 

$ 167.00 

$ 227.00 

35.9% 

Canceled 



ASO 

XTE or RXTE 

$ 100.00 

$ 373.00 

273.0% 

53 

48 

-9.4% 
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